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Executive  Summary 


The  Department  of  Defense  faces  many  challenges  related  to  deploying  new  systems  to  meet  the 
needs  of  the  warfighter.  Among  these  are  compressed  development  times  and  increased 
required  capabilities.  There  is  a  real  need  for  systems  engineering  expertise  in  the  acquisition 
enterprise  to  guide  the  technical  design,  development,  integration  and  testing  of  such  systems. 

At  the  same  time,  there  is  a  potential  skills  gap  due  to  an  aging  workforce,  many  of  whom  are 
near  retirement,  and  the  experience  development  curve  needed  to  bring  new  talent  up-to-speed. 
Often  this  experience  development  takes  many  years  and  spans  multiple  programs.  The  idea 
behind  the  Systems  Engineering  Experience  Accelerator  (SEEA)  is  to  accelerate  the  experience 
development  so  that  new  talent  can  become  proficient  much  more  quickly  to  meet  DoD  needs. 
This  is  done  by  using  educational  technology,  and  specifically  the  notion  of  role-playing  in 
immersive  environments  where  a  learner  not  only  leans  the  technical  skills  associated  with  a 
systems  engineering  position,  but  also  critical  skills  in  persuasion  and  decision-making  needed  to 
deploy  technical  skills  effectively. 

The  original  SEEA  was  developed  to  let  the  learner  play  the  role  of  a  chief  systems  engineer  in  an 
acquisition  program  for  a  new  unmanned  aerial  vehicle  (UAV)  system.  It  was  based  on  the 
concept  of  the  learners  assuming  this  role  shortly  after  preliminary  design  review  (PDR)  and 
seeing  it  through  until  after  the  start  of  low-rate  initial  production  (LRIP).  The  experience  was 
the  focus  of  an  initial  implementation.  However,  subsequent  work  focused  only  on  the  first  part 
of  the  experience,  the  time  from  the  PDR  until  the  critical  design  review  (CDR). 

This  project  has  taken  the  UAV  experience  and  improved  it  substantially  in  a  number  of  areas  in 
preparation  for  its  use  in  the  Defense  Acquisition  University  curriculum  for  systems  engineers. 
First,  several  new  capabilities  have  been  added.  These  include  a  trade  study  for  additional 
technical  content,  system  reliability  as  a  key  performance  parameter,  and  technical  debt  to  allow 
the  learner  to  see  the  effect  of  deferred  work.  Second,  a  number  of  improvements  have  been 
made  to  the  experience.  Most  importantly,  the  experience  lifecycle  has  been  updated  so  that 
the  work  completed  earlier  to  improve  the  PDR-to-CDR  portion  of  the  experience  has  been  done 
to  improve  the  remaining  parts  of  the  experience.  Third,  the  SEEA  application  has  been  made 
compliant  with  federal  law  on  accessibility  for  disabled  employees  and  others.  Fourth,  the 
research  hypothesis  that  the  SEEA  can  accelerate  learning  in  systems  engineering  has  been  tested 
via  its  use.  Finally,  support  has  been  provided  for  deployment  at  DAU. 
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Introduction 


Recent  years  have  seen  numerous  applications  of  educational  technology  in  a  variety  of  domains. 
One  such  application  involves  the  immersion  of  the  learner  in  a  simulated  environment  where 
lessons  from  a  particular  domain  are  taught.  This  approach  is  called  computer-based  experiential 
learning.  The  advantage  of  such  an  approach  is  that  it  not  only  teaches  the  material  of  interest, 
but  it  also  provides  a  context  for  the  learner  to  apply  the  new  knowledge,  thus  facilitating  skill 
development  and  potentially  promoting  interest  in  learning.  This  type  of  approach  has 
advantages  over  traditional  on-the-job  training,  as  well.  For  instance,  the  learner  can  be  allowed 
to  learn  lessons  from  failure  with  little  adverse  consequence.  In  addition,  once  the  educational 
technology  platform  is  available,  it  can  be  used  for  such  training  at  relatively  little  expense. 

Systems  engineering  includes  knowledge  from  a  variety  of  disciplines.  It  is  not  until  this 
knowledge  is  put  into  practice  in  an  integrated,  real  world  environment  that  a  systems  engineer 
can  develop  the  necessary  insights  and  wisdom  to  become  proficient.  In  the  workplace,  these 
learning  events  are  often  distributed  sparsely  over  time  such  that  an  engineer  may  only  see  a 
complete  system  life  cycle  over  a  period  of  several  years.  As  a  result,  the  maturation  time  from 
completion  of  formal  studies  to  becoming  seasoned  systems  engineer  is  unacceptably  long, 
particularly  when  contrasted  with  the  clock  speeds  of  today's  society  in  which  career  change  is 
the  norm  rather  than  the  exception.  Educational  technologies  hold  the  promise  of  providing 
customized  learning  exercises  based  on  real-world  situations  to  reduce  the  reliance  on  extensive 
on-the-job  training  that  is  the  hallmark  of  current  workforce  development.  The  references 
section  of  this  report  includes  numerous  research  efforts  that  explore  the  benefits  of  educational 
technology. 

These  benefits  of  educational  technology  motivated  the  design  and  development  of  the  Systems 
Engineering  Experience  Accelerator  (SEEA)  as  a  means  to  accelerate  the  learning  of  systems 
engineers.  The  SEEA  is  a  technology  platform  that  promotes  learning  in  cycles,  similar  to  the  Kolb 
learning  cycle  (Kolb,  1994).  The  learner  is  presented  with  a  situation.  Then  he  or  she  investigates 
issues  to  determine  problems  and  root  causes.  The  learner  can  make  decisions  or 
recommendations  to  address  the  problems,  and  those  are  input  to  the  simulated  world.  The 
simulated  world  advances  to  a  new  state  based  on  its  trajectory  and  the  learner  decisions,  and 
the  learner  can  see  the  impact  of  those  decisions.  A  new  cycle  then  starts.  The  SEEA  supports 
single-learner  and  multi-learner  modes,  along  with  varying  degrees  of  instructor  and  mentor 
interaction. 

Along  with  the  SEEA  technology  platform,  an  initial  experience  was  designed  and  developed.  In 
this  experience,  the  learner  plays  the  role  of  a  chief  engineer  in  a  program  that  is  developing  a 
new  unmanned  aerial  vehicle  (UAV)  system.  The  learner  assumes  this  role  shortly  after  the 
program  has  completed  what  appears  to  be  a  successful  preliminary  design  review  (PDR).  The 
learner  is  tasked  with  identifying  and  correcting  issues  in  the  program  so  that  it  operates  on 
schedule  and  within  budget,  and  so  that  the  system  meets  the  technical  requirements  specified 
for  key  performance  parameters  and  technical  performance  measures  with  the  desired  quality 
level.  The  experience  lasts  for  several  phases  corresponding  to  phases  in  a  DoD  acquisition 
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program,  with  each  phase  culminating  in  a  milestone  review.  The  last  phase  in  the  experience  is 
low-rate  initial  production,  where  the  learner  sees  the  results  of  the  program  design  and 
development  work. 

The  experience  was  initially  prototyped  and  demonstrated  at  Defense  Acquisition  University. 
Subsequent  work  then  focused  on  the  acquisition  phase  lasting  from  preliminary  design  review 
to  critical  design  review.  This  project  extends  the  work  done  on  that  particular  phase  of  the  UAV 
experience  in  a  number  of  ways.  The  UAV  experience  is  given  new  features  and  capabilities,  and 
the  work  done  on  the  PDR-to-CDR  phase  is  extended  to  the  remaining  phases.  The  experience 
has  also  been  upgraded  so  that  it  complies  with  the  Section  508  Amendment  to  the  Rehabilitation 
Act  of  1973.  This  law  states  that  federal  agencies  must  give  disabled  employees  and  members  of 
the  public  access  to  information  that  is  comparable  to  the  access  available  to  others.  Finally, 
learning  evaluation  and  deployment  of  the  SEEA  at  DAU  are  addressed. 

The  remainder  of  this  report  is  organized  as  follows.  The  research  overview  summarizes  the  goals 
to  be  achieved  by  the  research,  describes  the  baseline  SEEA  experience  developed  previously, 
and  outlines  the  tasks  performed  in  this  project.  Then  each  task  is  discussed  in  detail  along  with 
the  approach  taken  and  the  results.  These  include  new  experience  capabilities,  new  experience 
features.  Section  508  compliance,  validation  of  the  research  hypothesis,  and  support  for 
deployment  of  the  SEEA.  Finally,  the  conclusions  and  future  research  are  presented. 

Research  Overview 


This  project  addresses  fundamental  issues  in  the  effectiveness  of  systems  engineering  education 
and  training  in  meeting  DoD  and  societal  needs  in  the  area  of  complex  systems  design 
development.  This  section  discusses  the  project  goals,  a  background  of  the  existing  SEEA,  and 
the  tasks  performed  in  the  project  to  advance  the  SEEA  and  prepare  it  for  usage. 


Research  Goals 

Problem  Statement:  Traditional  Systems  Engineering  (SE)  education  is  not  adequate  to  meet  the 
emerging  challenges  posed  by  ever  increasing  Systems  and  Societal  demands,  the  workforce 
called  upon  to  meet  them  and  the  timeframe  in  which  these  challenges  need  to  be  addressed. 

Program  Goal:  Transform  the  education  of  SE  by  creating  a  new  paradigm  capable  of  halving  the 
time  to  mature  a  senior  SE  while  providing  the  skills  necessary  to  address  emerging  system's 
challenges. 

The  education  of  Systems  Engineering  needs  to  be  transformed  to  prevent  obsolescence  and 
maintain  relevancy.  We  postulate  that  the  new  paradigm  must  be: 

•  Experience  Based:  Providing  accelerated  learning  opportunities  through  experience- 
based  interactive  sessions  (Kolb,  1984). 
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•  Agile:  Allowing  for  quality,  timely  development  of  course  material  that  is  most 
appropriate  for  the  target  students. 

•  Integrated:  Provides  an  integration  point  of  multi-disciplinary  skills  and  a  wide  range  of 
Systems  Engineering  knowledge  in  a  setting  that  recreates  the  essential  characteristics  of 
the  practicing  environment. 

•  Lean:  Providing  the  greatest  amount  of  benefits  with  the  minimal  number  of  steps  and 
least  amount  of  effort. 

•  Leveraged:  Enabling  capability  growth  through  the  leveraging  of  computational  and 
information  technologies,  and  prior  Systems  work. 

•  Extensible:  Providing  the  capability  to  expand  and  enhance  capabilities  for  future  growth 
without  having  to  make  major  changes  in  the  infrastructure. 

•  Implementable:  Enabling  widespread  impact  through  economically  viable,  rapid 
development  and  deployment  of  educational  and  training  programs  for  participants  with 
multiple  levels  of  competence  and  background. 

The  high-level  goals  for  this  program  are  two-fold.  First,  the  results  of  this  research  should  result 
in  the  creation  of  a  new  means  of  maturing  Systems  Engineers  in  substantially  less  time  than 
normally  required  to  reach  a  senior  level  of  experience.  It  is  believed  that  creating  new 
experience-based  training,  which  is  integrated  with  and  complements  more  traditional  means  of 
knowledge  acquisition,  will  be  necessary  to  achieve  this  goal.  Secondly,  this  research  should 
provide  the  means  by  which  these  results  can  be  obtained  in  an  economically  attractive  manner. 


Existing  SEEA  Prototype 

The  SEEA  was  designed  and  developed  in  a  series  of  previous  projects.  This  work  created  the 
technology  platform,  as  well  as  an  initial  experience  focused  on  the  design  and  development  of 
a  new  unmanned  aerial  vehicle  system. 

The  SEEA  technology  provides  a  graphical  user  interface  allowing  the  learner  to  see  the  program 
status,  interact  with  non-player  characters  to  gain  additional  program  information,  and  make 
technical  decisions  to  correct  problems.  It  also  provides  the  capability  to  simulate  the  program 
into  the  future  based  on  these  learner  decisions,  so  that  outcomes  can  be  shown  to  the  learner. 
This  cycle  of  decision  and  simulation-into-the-future  supports  the  Kolb  cycle  of  experiential 
learning;  the  Experience  Accelerator  uses  multiple  such  cycles  operating  through  the  lifecycle  of 
the  program.  In  particular,  this  approach  allows  illustration  of  the  effect  of  upstream  decisions 
on  downstream  outcomes  in  the  system  lifecycle.  The  SEEA  can  support  a  wide  variety  of  systems 
domains  and  areas  of  expertise  through  changes  to  the  experience.  Multi-player  technology  has 
been  developed  to  allow  live  player  support  for  team-based  learning,  as  well  as  for  an  instructor 
and  mentor  to  provide  advice  and  feedback.  The  SEEA  contains  a  number  of  inter-connected 
modules  as  shown  in  Figure  1. 
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Experience  Accelerator  Block  Diagram 


Experience  Generic 


Experience  Specific 


Figure  1.  SEEA  architecture 

•  Experience  Master:  contains  the  overall  Experience  state  and  provides  control  and 
sequencing  for  the  other  major  EA  modules. 

•  Challenge  Control:  contains  the  Learner  profiles  and  Experience  history  logs  and 
leverages  these  in  conjunction  with  the  competency  taxonomy  and  'Aha'  moments  (i.e., 
generic  lessons  that  successful  and  experienced  systems  engineers  would  know,  such  as 
seemingly  unimportant  upstream  decisions  can  have  major  negative  downstream 
ramifications)  to  determine  the  appropriate  challenges  and  landmines  for  each  Learner. 

•  Simulation  Engine:  determines  the  future  state  of  the  system  and  outputs  to  be 
presented  to  the  Learner. 

•  Non-Player  Characters  (NPC)  Engine:  represents  other  non-player  characters  in  the 
simulation  and  creates  and  assembles  the  content  for  Learner  interactions,  and 

•  Presentation  Engine:  accepts  inputs  from  the  Learner  and  provides  the  presentation  of 
the  Experience  interface  to  the  Learner. 

The  SEEA  utilizes  a  cycle-based  approach  to  learning,  and  it  also  supports  multiple  phases,  each 
of  which  contain  learning  cycles.  In  an  acquisition  program,  phases  can  correspond  to  the  times 
in  between  major  milestone  reviews.  Each  phase  a  set  of  sub-phases  and  learning  cycles.  We 
can  also  consider  each  use  of  the  experience  by  a  learner  as  a  high-level  learning  cycle.  This 
concept  is  shown  in  Figure  2. 
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Learning  Cycles 


Figure  2.  Learning  cycles  and  phases 

The  UAV  experience  utilizes  the  phase  and  cycle  approach,  with  phases  shown  in  the  Figure  3 
overview  of  the  UAV  experience.  The  first  two  phases  correspond  to  familiarization  with  the 
SEEA  and  introduction  to  the  UAV  exercise.  In  the  new  employee  orientation,  the  learner 
assumes  the  role  of  the  chief  systems  engineer  of  the  UAV  program.  It  has  just  passed  PDR.  The 
learner  then  goes  through  a  pre-integration  phase,  culminating  with  critical  design  review.  Next 
comes  the  integration  phase,  which  ends  with  the  flight  readiness  review  (FRR).  The  field  test 
phase  is  next,  and  it  ends  with  the  production  readiness  review  (PRR).  In  the  limited 
production/deployment  phase,  the  learner  sees  the  results  of  the  acquisition  program  in  terms 
of  systems  being  produced  in  low-rate  initial  production.  The  experience  ends  with  summary 
statistics,  and  in  a  reflection  phase  the  learner  is  presented  with  his  or  her  decisions  throughout 
the  experience  compared  to  what  are  judged  to  be  the  correct/best  decisions. 

The  UAV  system  is  divided  into  three  sub-systems  being  developed  by  different  sub-contractors, 
plus  a  prime  contractor  overseeing  the  overall  system  design,  development  and  integration.  The 
three  sub-systems  are:  the  airframe  &  propulsion  system,  the  command  &  control  system,  and 
the  ground  station  system. 
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Context: 

•  Based  on  SME  interviews  and  a  systems  dynamics 
model  of  large  system  acquisition  and  development 

•  Complex  experience  that  covers  the  entire  lifecycle 


USDoDUAV  System 
Acquisition: 

•  Prime  Contractor - 
System 

•  Subsystem  1  - 
Airframe  and 
Propulsion 

•  Subsystem  2  - 
Command  and 
Control 

•  Subsystem  3  - 
Ground  Support 

•  Subcontract  for 
each  subsystem 


Phases: 

■  EA  Introduction 

■  Phase  0:  New  Employee 
Orientation 

■  Experience  Introduction 

■  Phase  1:  New 
Assignment  Orientation 

■  Experience  Body 

■  Phase  2:  Pre-integration 
System  Development -> 
CDR 

■  Phase  3:  Integration -> 
FRR 

■  Phase  4:  Field  Test -> 
PRR 

■  Phase  5:  Limited 
Production/Deployment 

■  Phase  6:  Experience  End 

■  Experience  Conclusion 

■  Phase  7:  Reflection 


UAV  KPMs: 

Schedule,  Quality,  Range,  Cost 


Figure  3.  UAV  experience  overview 

During  the  phases,  the  learner  executes  multiple  learning  cycles.  Each  cycle  is  characterized  by 
a  presentation  of  updates  to  the  system  state,  followed  by  the  learner  making  a  set  of 
recommendations  bbto  correct  any  perceived  problems  or  issues.  The  SEEA  provides  a  form  into 
which  the  recommendations  are  placed.  A  recommendation  may  be  in  the  form  of  a  drag 
reduction  program,  for  example,  if  the  air  vehicle's  drag  is  too  high.  This  recommendation  would 
come  with  a  target  value  for  drag.  The  learner  can  make  technical  performance 
recommendations  or  programmatic  decisions  (hire  additional  engineering  staff  or  shift 
resources). 

These  recommendations  are  fed  into  the  simulation  module,  which  is  responsible  for  maintaining 
the  state  of  the  simulated  world.  This  module  implements  a  large-scale  system  dynamics 
simulation  model  to  perform  this  function.  The  simulation  module  executes  for  a  period  of  time, 
reflecting  3-6  months  of  the  simulated  world  and  allowing  the  learner's  decisions  to  play  out 
during  the  time.  The  learner  then  is  presented  with  an  updated  state. 

The  learner  may  explore  this  state  in  two  main  ways.  First,  the  simulation  module  produces  a  set 
of  graphs  that  reflect  the  values  of  key  performance  measures  that  impact  technical 
performance,  cost  and  schedule.  The  learner  may  also  interact  with  non-player  characters,  ask 
questions  to  them,  and  respond  to  their  concerns.  The  NPCs  use  condition-based  text  dialog  for 
simulated  emails  and  phone  conversations.  The  learner  may  select  questions  to  ask  on  different 
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topics  from  a  menu,  and  the  NPCs  respond  with  dialog  that  incorporates  the  system  state.  For 
instance,  if  drag  is  a  problem,  the  NPC's  dialog  would  be  customized  to  reflect  that  situation  if 
the  learner  asks  about  drag,  and  the  NPC  may  also  provide  clues  about  root  causes  or  potential 
solutions.  NPCs  include  the  program  manager,  the  lead  systems  engineer  of  the  prime 
contractor,  the  previous  chief  engineer  for  the  UAV  program,  a  test  &  evaluation  representative, 
and  a  mentor.  The  mentor  provides  guidance  to  the  learner. 

The  learning  cycle  details  are  shown  in  Figure  4.  The  learner  can  explore  the  simulated  world  by 
viewing  a  chart  (a  chart  for  drag  is  shown),  interacting  with  NPCs,  or  viewing  the  program 
dashboard  which  provides  a  high-level  view  of  program  status  with  respect  to  technical 
performance,  cost,  schedule  and  quality.  Then  the  learner  enters  recommendations  into  the 
recommendation  form.  The  recommendation  form  provides  updated  variable  values  to  the 
simulation  module,  which  executes  and  updates  the  simulated  world. 


Simulated  world 


Simulation 

module 


Figure  4.  Learning  cycle  actions 

The  learner's  score  is  assessed  by  how  well  technical  performance,  cost,  schedule  and  quality 
metrics  result  at  the  end  of  the  experience.  If  a  poor  job  is  done,  it  is  possible  that  the  learner 
will  be  terminated  from  the  program. 

The  SEEA  and  the  UAV  experience  have  been  used  at  Defense  Acquisition  University  and  the 
University  of  Alabama,  Huntsville.  The  SEEA  is  implemented  as  a  thin-client  web  application, 
which  has  newly  been  transitioned  from  Flash  to  FITML5.  The  learner  needs  only  a  browser  to 
use  the  application  which  is  discussed  in  the  deployment  guide  section  of  this  report.  The  FITML5- 
based  login  screen  is  depicted  in  Figure  5. 
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O  «a  E3  Experience  Accelerator:  X  -f 


O  I  107.170.70.1 1 1/*!/login  -fa  =~  tiL  \& 


SERC 


SERC  Systems  Engineering  Experience  Accelerator 


SYSTEMS 

ENGINEERING 


Please  login 
Username: 


Password: 


Login 


£AU 

Defense  Acquisition  University 


UAS  XV-5  Acquisition  Lead  Systems  Engineer 
Experience 


EA  Version  0.5.32 
Server  Status:  Online 


About  this  experience  |  About  EA 

Copyright  ©  Experience  Accelerator  Project  201 7 


a  ■  «i  >■  -i  ®  a  » 


Figure  5.  SEEA  login  screen 


Summary  of  Research  Tasks 

In  the  prior  increments,  this  research  has  established  a  concept  and  architecture  for  the  SE 
Experience  Accelerator,  developed  an  evolving  prototype,  and  piloted  that  prototype  in  DAU 
classes  and  industry  workshops  to  refine  the  concepts  and  improve  the  implementation.  The 
Experience  Accelerator  has  continued  to  mature  and  now  has  a  variety  of  capabilities  that  should 
support  experiences  in  numerous  domains  and  in  several  different  single  and  multi-player 
modes.  There  is  great  potential  for  the  EA  technology,  experiences  and  tools  in  a  wide  range  of 
areas  and  domains. 

The  focus  of  Increment  4  is  in  four  major  areas.  This  first  is  to  increase  the  scope  of  the  SEEA 
beyond  programmatics  to  include  trade-studies  which  are  a  vital  part  of  a  systems  engineer's  job. 
The  second  is  to  improve  the  features  and  capabilities  of  the  SEEA  in  a  number  of  important 
areas.  Some  of  these  include  updating  the  experience  through  initial  production  and  the 
inclusion  of  technical  debt  and  reliability.  The  third  area  is  to  validate  the  research  hypothesis 
with  an  evaluation  of  the  learning  that  occurs  through  SEEA  use.  Finally,  there  is  the  effort 
required  to  support  the  SEEA  in  the  DAU  environment.  Below  is  a  description  of  the  work 
objectives  in  each  area. 
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1.  Task  1.  New  Experience  Capabilities 

The  current  DAU  experience  is  primarily  programmatic  in  its  focus.  However,  systems  engineers 
also  need  the  capability  to  perform  trade  studies  and  make  technical  decisions.  As  such,  these 
capabilities  need  to  be  added  to  enhance  the  current  DAU  experience  to  test  the  hypothesis  in 
these  areas.  The  following  summarizes  the  new  EA  capabilities: 

•  Tl.l:  Add  trade  study  and  technical  decision  making  activities.  Introduce  and  validate  new 
Actuation  System  Re-architecture  (Fontaine  Master's  Thesis)  trade  study.  Incorporate 
necessary  artifacts,  dialogs,  and  simulation  capabilities  (using  new  tools) 

•  T1.2:  Add  reliability  KPP  and  add  technical  debt  accumulated  in  one  phase  to  be  worked 
in  the  next  phase. 

2.  Task  2.  Experience  Improvements 

In  addition  to  the  new  capabilities,  a  series  of  improvements  in  the  current  experience  need  to 
be  made  to  ensure  that  the  user  of  the  EA  can  effectively  learn  the  desired  lessons.  The  current 
DAU  experience  can  be  extended  in  both  scope  and  new  capabilities.  The  experience  was 
designed  to  support  the  UAV  project  from  PDR  to  limited  production.  Past  work  addressed  the 
first  phase  of  the  experience,  namely  up  until  the  CDR.  Four  additional  phases  have  been 
prototyped.  However,  these  additional  four  phases  have  not  kept  up  with  the  development  of 
the  PDR-to-CDR  phase  which  became  the  focus  of  the  DAU  experience.  The  following  research 
activities  support  these  improvements: 

•  T2.1:  Update  simulation,  dialog  and  artifacts  to  be  consistent  with  the  current  pre¬ 
integration  phase  throughout  the  complete  the  full  life-cycle. 

•  T2.2:  Tune  simulation  per  sponsor  and  pilot  feedback. 

•  T2.3:  Complete  the  NPC  role  of  mentor  to  better  fit  DAU  course  and  provide  instruction. 

•  T2.4:  Update  Lecture/Student  Materials  to  facilitate  learning  evaluation  using  logged  data 
from  the  EA  leveraging  work  from  learning  evaluations. 

3.  Task  3.  Section  508  Compliance 

In  the  planning  stage  of  this  project,  the  objective  is  to  determine  the  requirements  for  simulated 
learning  technologies,  specifically  the  EA,  under  Section  508.  Based  on  these  requirements,  the 
next  step  is  to  determine  a  compliance  strategy  for  design,  implementation  and  validation,  and 
then  provide  an  estimate  for  this  work.  The  objective  for  Implementation  part  is  to  implement 
these  changes  to  ensure  that  the  EA  is  Section  508  compliant.  The  necessary  migration  of  the 
Presentation  Engine  from  Flash  to  HTML5  is  being  accomplished  in  the  EA  Tools  Part  3  RT.  The 
work  in  this  project  is  composed  of  the  following  major  activities: 

•  T3.1:  Determine  methods  for  evaluating  Section  508  compliance,  authoring  content  and 
developing  simulated  learning  applications. 

•  T3.2:  Determine  necessary  changes  for  Section  508  compliance. 
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•  T3. 3:  Translation  of  EA  screens  (login,  multi-player,  desktop,  dashboard,  etc.)  to  compliant 
formats. 

•  T3.4:  Translation  of  EA  artifacts  (documents,  email,  recommendation  forms,  charts, 
dialog)  to  compliant  formats. 

•  T3.5:  Evaluation  of  508  compliance. 

4.  Task  4.  Validate  Research  Hypothesis  -  Learning  Evaluation 

The  validation  of  the  research  hypothesis  through  the  learning  evaluation  process  requires 
support  for  classroom  instructors,  students,  and  the  active  capture  and  analysis  of  learning 
results.  These  evaluations  will  take  place  for  the  DAU  experience  at  DAU  and  potentially  other 
institutions.  Learning  evaluation  is  a  critical  element  to  validating  the  research  hypothesis  of  the 
SEEA  relative  to  accelerating  learning.  In  addition  to  the  research  in  Increment  4,  research  in 
learning  efficacy  is  taking  place  through  a  doctoral  dissertation  funded  by  ARDEC,  educational 
research  taking  place  at  Stevens,  and  tools  development  funded  by  SERC  core  funds.  However, 
research  needs  to  be  done  to  determine  specifically  how  learning  will  be  evaluated  for  the  DAU 
experience. 

The  following  are  the  items  that  will  need  to  be  completed: 

•  T4.1:  Provide  support  to  DAU  and  potentially  other  instructors  with  EA  application, 
materials  and  staff. 

•  T4.2:  Support  pilot  uses  of  the  EA  at  DAU  and  potentially  other  institutions. 

•  T4.3:  Design  learning  evaluation  and  collect  learning  data. 

•  T4.4:  Analyze  data,  evaluate  learning  effectiveness,  and  write  report  of  results. 

5.  Task  5.  Support  DAU  EA  Deployment 

Additional  work  is  necessary  to  create  a  plan  that  is  specific  to  DAU  needs.  Depending  on  the 
path  that  DAU  selects,  this  may  require  training  of  DAU  staff  and  the  development  of  additional 
support  materials.  The  following  are  the  items  completed: 

•  T5.1:  Specification  of  the  concrete  deliverables  so  that  the  EA  can  be  used  in  the  DAU  on 
a  long-term  basis.  This  includes  criteria,  requirements,  sustaining  support,  technical 
details  of  the  hosting  requirements. 

•  T5.2:  Determine  Hosting  Solution(s)  for  DAU:  Options  include  commercial  support 
(external  network),  DAU  support  (DAU  network)  and/or  support  on  a  laptop  within  the 
classroom  (classroom  network). 

•  T5.3:  Migrate  Development  to  3rd  Party  Support:  This  includes  identifying  support  staff 
and  cost,  and  determining  process  and  procedures  for  ticket  creation  and  processing. 

•  T5.4:  Update  EA  documentation:  This  includes  the  concept  of  operations,  system  and 
support  specifications,  architecture  and  design  document,  and  experience  tools  and 
development  document. 

•  T5. 5:  Support  of  deployment  plan. 
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Task  1  -  New  Experience  Capabilities 


To  provide  more  focus  on  the  technical  aspects  of  an  acquisition  program,  two  categories  of  new 
capabilities  were  added  to  the  UAV  experience.  Based  on  a  student  project  at  Stevens  Institute 
of  Technology  (Fontaine,  2015),  a  trade  study  was  the  first  new  capability  added  to  the 
experience.  In  addition,  system  reliability  was  added  as  a  new  key  performance  parameter,  and 
technical  debt  was  also  added  so  that  the  learner  can  understand  what  occurs  when  work  in  a 
design  and  development  effort  is  deferred. 


T echnical  Decision-Making  via  Addition  of  Learner  T rade  Study 

The  trade  study  is  implemented  in  the  Experience  Accelerator  as  a  new  capability.  The  trade  study 
focuses  on  leading  the  learner  through  a  recently  evolved  issue  that  requires  the  leaner  to 
conduct  a  trade  study  for  choosing  the  best  actuator  for  the  situation. 

A  high-level  flow  of  the  trade  study  cycle  has  been  established  to  serve  as  a  storyboard  upon 
which  the  simulation  events,  interactions,  and  recommendation  can  be  built.  The  major  activities 
in  the  trade  study  cycle  are  shown  in  Figure  6  below. 


Trigger  Trade 

Call  from  Tom  Williams. 

Discussion  with  Col.  Rogers 
Follow-up  with  Tom  Williams 

Setup 

Identify  candidate  architectures 

(Optional)  Discussion  with  Mentor  for  any  coaching 

Establish  Criteria 

Collect  Data 

Provide  Learner  with  technical  reference  material 
Analyze  /  Decide  /  Report 

Populate  and  evaluate  trade  matrix 
Document  rationale  and  communicate  to  PM 

Simulation 

Update  model  to  new  state 
Feedback  to  Learner 


Figure  6  T rade  Study  Cycle  Flow 

Two  alternative  approaches  to  triggering  the  trade  study  were  considered  -  allowing  it  to  occur 
during  any  one  of  the  Pre-CDR  cycles,  or  locking  it  into  a  specific  cycle.  For  the  purposes  of  the 
prototype,  which  will  be  used  as  a  baseline  for  evaluating  the  EA's  effectiveness  as  a  teaching 
tool,  the  design  team  determined  that  consistency  would  be  favored  over  a  more  realistic,  but 
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less  deterministic,  approach.  At  this  time,  the  trade  study  has  been  integrated  into  Phase  2A, 
Cycle  4.  It  is  enabled  by  a  server-side  setting. 

The  trade  is  triggered  in  conjunction  with,  and  signaled  by,  a  step  change  in  system  weight.  From 
a  storyline  perspective,  this  is  explained  as  a  consequence  of  needing  to  add  shielding  and 
filtering  to  the  aircraft  wiring  harnesses  to  meet  stringent  shipboard  electromagnetic 
interference  (EMI)  and  electromagnetic  compatibility  (EMC)  requirements.  This  is  a  common 
concern  for  many  military  systems.  Introducing  the  trade  study  in  this  fashion  provides  an 
opportunity  to  make  the  learner  aware  of  this  real  world  issue. 

The  trade  study  starts  with  special  dialogues  with  the  PM,  Mentor  and  the  Prime-PSE  as  shown 
in  Figure  7.  The  Learner  will  receive  information  and  comments  from  PM  and  Prime-PSE.  The 
Mentor  will  provide  the  Learner  more  detailed  information  on  how  to  conduct  a  trade  study. 
Once  the  trade  scope  has  been  established,  technical  reference  material  and  interactive  NPC 
dialogues  are  used  to  inform  the  Learner  of  the  strengths  and  weaknesses  of  each  of  the 
alternatives.  This  information  serves  as  the  basis  for  their  recommendation. 

Once  the  Learner  has  reviewed  the  information,  they  will  complete  a  trade  study 
recommendation  form.  Based  on  the  background  information  and  dialogues,  they  will  determine 
appropriate  values  for  the  relative  criteria  rankings,  score  each  of  the  alternatives  against  the 
criteria,  review  the  resulting  trade  scoring  (as  determined  by  the  calculations  in  the  form),  and 
evaluate  the  sensitivity  of  the  results  against  changes  to  the  inputs.  They  will  then  document 
their  recommended  architecture  and  rationale  and  submit  the  recommendation  form  to  the  PM. 


SERC  Home  Dashboard  Recommendation  Form  UDrary 


Phone  Calls 

Click  to  call: 

£] 

I 

A 

I 

f, 

A 


Tandy  Rogers 


Tom  Williams 

Prune  Lead  Systems 

Engineer 

Onane 

Dale  Masters 

ExlSE 


Chris  Wilson 

Mentor 

Online 


Linda  Jackson 

Government  Test  and 
Evaluation 


Jett  Rossman 

DAU  HR  manage 
Not  Available 


Call  Panel: 


Col  Tandy  Rogers  Rogers  -  how  can  I  help  you? 

Learner  Good  morning  we  nave  some  troubling  news  from  tne  XZ-S  pnme  i  was  just  informed  that  the 
more  aggressive  EMI  requirements  add  components  that  push  tne  weight  budget  more  man  10%  over 
expected 

Col  Tandy  Rogers  <D>  Expletive  deietedi  </b»  Not  acceptable  We  need  to  reassess  Have  tne  folks  at 
UAventure  provided  any  suggestions  to  mis  minor  barricade? 

Learner  UAventure  has  proposed  changing  me  design  for  the  hydraulic  actuators  to  some  other  technology 
•  electro- hydrostatic,  combination  ot  hydraukc  and  electro-mechanical  or  all  electnc 

Col  Tandy  Rogers  Right  Sorry  about  that  it's  been  a  trying  day  all  around  What  do  you  think  about  those 
options? 


learner  I  think  tney  seem  reasonable 


Tom  Williams  has  been  pretty  up  front  about  this,  although  It  is 
i  proposal  tor  a  fix  However  mere  are  a  lot  ot  factors  that  need  to  be 


Col  Tandy  Rogers  Good  i  need  to  enforce  mat  weight  s  King  given  Os  impact  on  fuel  capacity  and  range 
but  we  cam  lose  sqm  otothei  possible  impacts  of  tne  change  We  also  need  to  be  consoous  about  pricing 
oursetves  mio  cancelation  now  or  later  </D>ir  we  come  up  win  the  ngnt  design  the  sponsor  could  be 
understanding  of  a  minor  schedule  mt  but  we  need  to  tread  carefUffy  there 


Figure  7  Trade  Study  Dialogs  with  PM 
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For  the  prototype  EA,  the  actuation  system  alternatives  are: 


•  (Baseline)  Hydraulic  actuators  -  This  consists  of  an  engine  driven  hydraulic  pump 
providing  distributed  hydraulic  power  to  all  of  the  actuators. 

•  Electrohydrostatic  actuators  -  In  this  configuration,  each  actuator  uses  distributed 
electrical  power  to  drive  localized  hydraulic  rams. 

•  Split  electromechanical/electrohydraulic  -  This  is  a  hybrid  combination  of 
electromechanical  and  electrohydraulic  actuators,  balancing  the  actuation  needs  of  each 
location  and  function  to  optimize  overall  system  weight. 

Electromechanical -This  configuration  is  purely  electromechanical  in  nature,  allowing  for 
a  major  reduction  in  the  size  of  the  vehicle's  hydraulic  system. 


For  the  trade  prototype,  these  four  alternative  architectures  are  evaluated  against  the  following 
evaluation  criteria: 

o  Non-recurring  Engineering  (NRE)  -  This  represents  the  additional  design  and 
development  costs  associated  with  a  given  architecture. 

o  Flyaway  Cost  -  This  represents  any  impact  to  the  final  cost  of  a  delivered  system. 

o  Total  System  Weight  Impact  -  This  includes  the  weight  increases  and  decreases  to  all  of 
the  various  aircraft  subsystems. 

o  Schedule  -  This  is  a  measure  of  impact  to  the  system  development  timeline. 

o  Thermal  -  This  criteria  represents  impacts  to  the  thermal  performance  of  the  UAV, 
specifically  with  respect  to  hot  operating  temperatures.  It  is  included  to  highlight  the 
criticality  of  thermal  management  for  electromechanical  actuators. 

o  EMI/EMC-This  highlights  the  differences  between  hydraulics  and  electric  actuators  with 
respect  to  emissions  and  susceptibility. 

o  Reliability  -  A  typical  SE  metric.  Reliability  will  differentiate  the  architectures  from  the 
perspectives  of  complexity  and  redundancy. 

Integration  Risk  -  This  represents  the  level  of  complexity  and  associated  risk  for 
incorporating  a  given  architecture. 


As  shown  in  Figure  8,  the  learner  will  be  presented  by  a  Microsoft  Excel  style  work  sheet  where 
the  leaner  can  input  different  parameters  and  weights  for  actuators'  attributes  and  then  decide 
on  which  actuator  technology  to  choose.  The  learner  can  either  work  on  the  online  form  or 
download  it  as  an  Excel  file  to  work  offline.  After  sending  the  recommendation  to  PM,  the  learner 
will  go  back  to  routine  UAV  experience  with  regular  recommendation  forms.  The  learner  can  save 
the  Excel  file  and  then  submit  them  to  the  instructor  for  comments  or  assessment.  Figure  9  below 
shows  the  design  of  the  worksheet  with  reference  values  and  inputs. 


Report  No.  SERC-2017-TR-167 


14 


August  8,  2017 


Dashboard  Recommendation 

Library  Graphs  FAQ  2  Phone  &  Emails 

♦  Fast 

Forward  ^Notification 

0  G*  Quel 

Trade  Study  Recommendation  Form 

Criteria  -> 

NRE  Flyaway  Cost  T0UI  SystemWeight  Schedule  Thermal  EMI/EMC  Rellabdity  Integration  Risk 

Weighting  (110  Peiatlvel 

Weighting  % 

1 

1 

* 

I 

1 

1 

1 

1 

13* 

13* 

13* 

13* 

13* 

13* 

13* 

13* 

Architecture  Options 

NRE  Flyaway  Cost  Total  System  Weight  schedule  Thermal  EMI/EMC  Reltabdlty  Integration  Risk 

1  -  Electrohydraulic  actuators 

2  -  Electrohydrostatic  actuators 

3  -  Split  Electromech/electrohydrau  >c 

4 -Electric  actuators 

Tr»d«  Rk  Form 


Select  Recommended  Architecture  1  -  Electrohydraulic  actuators 
Rationale 
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Weight  savings  alone  are  not  enough  to  justify  the  drastic  increase  in  technical  risk,  cost,  schedule,  environmental  compliance,  etc. 

Continuing  with  the  baseline  design  is  the  most  appropriate  course  of  action  for  this  program. 

Figure  9  Trade  Study  Worksheet  Design 

The  trade  study  introduces  technical  issues  into  the  experience  that  were  not  there  previously. 
The  different  alternatives  impact  several  performance  factors  in  particular.  Two  of  these  have 
been  added  to  the  experience  -  actuator  reliability  and  thermal  management.  A  technical 
performance  measure  for  actuation  system  reliability  is  added.  Under  the  baseline  actuator 
system,  the  reliability  is  acceptable  (above  the  minimum  threshold)  and  it  improves  over  time  to 
reach  a  steady  state  value.  If  one  of  the  other  alternatives  is  selected,  the  steady  state  target 
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value  can  be  changed,  thus  causing  the  reliability  to  drop  and  perhaps  become  unacceptable.  A 
graph  of  actuator  reliability  over  time  for  the  baseline  actuator  system  is  illustrated  in  Figure  10. 
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Figure  10.  Actuator  reliability  over  time 


Heat  rejection  and  heat  storage  are  two  metrics  introduced  for  thermal  management.  Heat 
rejection  refers  to  the  amount  of  heat  created  by  the  actuators  that  can  then  be  dissipated  safely. 
We  consider  two  cases  in  terms  of  heat  created  -  nominal  and  peak.  Nominal  refers  to  heat 
created  under  normal  operating  conditions.  We  consider  that  there  is  a  maximum  threshold  that 
the  actuators  can  dissipate  (in  watts).  In  the  baseline  actuator  system,  the  nominal  heat  created 
is  below  this  threshold.  Thus,  the  actuators  can  operate  safely.  The  peak  is  above  the  threshold, 
however.  This  means  that  heat  storage  capacity  is  needed  to  trap  the  excess  heat  during  peak 
loads.  We  assume  that  a  peak  load  will  last  at  most  five  minutes.  The  heat  storage  required  then 
is  the  excess  heat  (peak  heat  created  minus  threshold  that  can  be  dissipated)  over  that  five 
minutes.  The  heat  created  per  time  unit  is  in  watts,  and  the  heat  storage  needed  is  then  in 
kilojoules.  If  the  peak  heat  created  is  too  high  relative  to  the  maximum  dissipated,  the  heat 
storage  required  can  exceed  the  heat  storage  maximum.  In  the  baseline  actuator  system,  the 
heat  storage  needed  is  less  than  the  maximum  allowed.  Heat  rejection  and  heat  storage  for  the 
baseline  actuator  system  are  depicted  in  Figure  11. 


TPM:  Reliability  -  Actuators  (current  phase) 
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TRM:  Actuator  Heat  Rejection  (current  phase) 


TPM:  Actuator  Heat  Energy  Storage  (current  phase) 


Figure  11.  Heat  rejection  and  storage  for  actuators 

If  the  electric  actuators  are  chosen  in  the  trade  study,  there  is  a  small  negative  effect  on  thermal 
management,  and  the  target  values  for  nominal  and  peak  heat  created  can  be  adjusted  to  reflect 
this.  If  the  electrohydrostatic  alternative  is  chosen,  there  is  a  large  negative  effect,  and  the 
targets  would  be  adjusted  to  ensure  that  the  nominal  heat  created  is  above  the  rejection 
threshold,  and  that  the  peak  heat  created  makes  the  heat  storage  needed  above  the  maximum 
allowed. 


Reliability  KPP  and  Technical  Debt 

The  current  UAV  experience  specified  vehicle  range  as  the  single  key  performance  parameter 
(KPP).  This  KPP  was  supported  by  a  number  of  technical  performance  measures  (TPMs)  that 
affect  range,  such  as  vehicle  weight,  drag  and  propulsion  efficiency.  System  reliability  has  been 
added  as  an  additional  KPP. 

System  reliability  is  defined  as  the  mean  time  to  failure  (MTTF).  As  MTTF  increases,  reliability 
increases.  Failure  is  affected  by  many  different  factors  relating  to  sub-systems  and  components. 
For  purposes  of  the  reliability  KPP  in  the  SEEA,  we  chose  to  limit  this  to  just  catastrophic  failures 
where  a  vehicle  would  be  lost.  In  addition,  we  limited  the  number  of  underlying  sub-systems 
that  could  affect  the  reliability  to  three.  Thus,  a  catastrophic  failure  in  one  of  the  following  three 
sub-systems  would  result  in  a  catastrophic  system-level  failure. 

•  Actuators  -  the  new  trade  study  focuses  on  the  actuators.  If  the  UAV  design  is  shifted  to 
a  new  architecture  for  actuators,  there  will  be  an  effect  on  the  actuator  reliability.  Thus, 
a  learner  decision  may  adversely  affect  system-level  reliability  so  that  it  becomes 
unacceptable. 

•  Uplink  -  the  uplink  brings  in  the  ground  station  sub-system  as  part  of  the  system-level 
reliability.  If  the  uplink  fails,  communication  cannot  be  sent  to  the  vehicle,  and 
presumably  it  will  be  lost. 


Report  No.  SERC-2017-TR-167 


17 


August  8,  2017 


•  Command  &  Control  System  -  the  command  &  control  system  clearly  is  critical  to  vehicle 
viability. 

In  the  baseline  case  (i.e.,  with  no  learner  decisions),  the  reliability  of  each  sub-system  improves 
until  it  reaches  a  steady  state.  Currently,  the  learner  recommendation  form  does  not  provide 
alternatives  that  impact  reliability  outside  the  actuator  architecture  decision,  although  that  could 
be  changed  in  the  future. 

The  system  reliability  is  determined  as  a  function  of  the  reliability  of  the  three  sub-systems. 
Letting  Ra  be  the  reliability  of  the  actuators,  Ru  be  the  reliability  of  the  uplink,  and  Rc  be  the 
reliability  of  the  command  &  control  sub-system,  we  can  define  the  reliability  of  the  system  Rs 
as: 


R 


S 


1  1  1\ 

Ra  Ru  Rj 


Thus,  as  each  sub-system  reliability  improves,  the  overall  system  reliability  improves.  Figure  12 
shows  the  reliability  over  time  of  the  actuators  and  that  of  the  overall  system  during  the  PDR-to- 
CDR  phase  of  the  program. 


[—Reliability  (MTBF)  —  Reliability  (MTBF)  Minimum  Threshold  —  Plan |  |—  Refablt.  IMtbf:  —  «an| 


Figure  12.  Reliability  over  time 

In  addition,  technical  debt  has  been  added  as  a  feature.  Technical  debt  is  deferred  work  that 
must  be  completed  at  a  later  date,  usually  at  some  sort  of  penalty.  Deferred  work  can  interfere 
with  work  scheduled  for  a  future  period.  In  addition,  it  can  be  more  costly  to  perform  in  the 
future  since  some  tasks  may  need  to  be  re-learned. 

We  apply  technical  debt  to  the  work  that  is  completed  to  ready  the  design  or  prototype  UAV  for 
a  milestone  review.  This  work  is  judged  complete  by  whether  or  not  it  meets  certain  entrance 
criteria  for  those  reviews.  The  SEEA  tracks  the  progress  toward  meeting  various  entrance  criteria 
for  critical  design  review,  flight  readiness  review,  and  production  readiness  review.  For  instance. 
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the  CDR  entrance  criteria  include  such  items  as  completion  of  digital  design  drawings,  structural 
load  analysis,  and  sub-system  design  and  verification  for  each  of  the  sub-systems. 

In  the  previous  version  of  the  UAV  experience,  which  focused  only  on  the  PDR-to-CDR  phase,  the 
learner  could  be  approved  to  proceed  to  CDR  without  meeting  100%  of  each  of  the  requirements. 
Each  of  the  entrance  criteria  was  set  with  a  certain  minimum  threshold  (e.g.,  90%  or  95%)  that 
needed  to  be  met.  This  threshold  could  be  varied  to  make  the  experience  more  or  less  difficult. 
Since  the  experience  ended  with  CDR,  there  was  no  penalty  for  the  small  percentage  of  work  left 
undone. 

However,  with  the  extension  of  the  experience  to  the  remaining  phases,  there  was  a  need  for 
that  unfinished  work  to  be  completed  in  the  next  phase.  This  is  accomplished  through  the 
concept  of  technical  debt. 

Each  of  the  milestone  reviews  has  six  to  eight  entrance  criteria.  Having  to  track  the  entrance 
criteria  progress  on  the  current  phase  plus  the  technical  debt  for  potentially  all  of  the  criteria  in 
the  previous  phase  could  be  overly  complex  for  the  learner.  Thus,  the  technical  debt  for  each 
sub-system  (and  corresponding  sub-contractor)  is  rolled  into  one  figure  that  is  a  weighted 
average  of  the  remaining  work  on  each  entrance  criteria  for  that  sub-system  and  the  percent 
effort  applied  to  that  entrance  criterion.  The  technical  debt  is  then  expressed  as  a  percentage  of 
the  overall  entrance  criteria  work  to  be  performed  by  the  sub-contractor.  Letting  TDi  be  the 
technical  debt  for  sub-contactor  i,  WRtjbe  the  percent  work  remaining  on  entrance  criterion  j 
by  sub-contractor  i,  and  be  the  percent  effort  applied  to  entrance  criterion  j  by  sub¬ 
contractor  i,  and  be  the  number  of  entrance  criteria  assigned  to  sub-contractor  i, 

ni 

TDi  =  ^  WRijWij 

i= i 

The  simulation  is  programmed  so  that  the  workforce  for  each  sub-contractor  focuses  on  technical 
debt  at  the  beginning  of  each  phase  before  work  is  done  on  the  entrance  criteria  for  that  phase. 
Once  the  technical  debt  work  has  been  finished,  work  ramps  up  on  the  entrance  criteria  for  the 
current  phase.  This  of  course  has  the  effect  of  delaying  the  completion  of  those  entrance  criteria. 
In  addition,  there  can  be  an  interest  factor  applied  to  the  technical  debt  such  that  the  amount  of 
work  is  increased,  simulating  the  additional  work  that  must  be  done  since  the  workforce  is  geared 
toward  the  current  phase  work,  not  that  of  the  previous  phase. 

Of  course,  the  technical  debt  has  an  effect  on  the  EVM  for  the  current  phase.  The  actual  cost  of 
work  performed  contains  cost  of  performing  the  technical  debt  up  until  such  time  as  that  work 
has  been  completed.  Then  it  shifts  to  the  current  work.  The  budgeted  cost  of  work  performed 
will  indicate  that  the  project  falls  behind  schedule  since  the  current  work  is  being  temporarily 
deferred.  To  show  the  learner  how  much  of  the  ACWP  is  funding  technical  debt,  a  new  graph  is 
added.  This  graph  tracks  the  ACWP  of  the  technical  debt  for  the  whole  program  and  compares 
it  to  the  remaining  budget  from  the  previous  phase.  If  there  is  not  any  budget  remaining,  then 
the  remaining  budget  is  zeroed  out.  Figure  13  shows  the  cost  of  technical  debt  from  the  PDR-to- 
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CDR  phase  (Phase  2)  that  is  incurred  in  the  integration  phase  (Phase  3).  Approximately  $26 
million  is  left  over  from  Phase  2,  with  the  assumption  that  those  funds  are  available  for  the 
technical  debt  work-off.  The  technical  debt  has  erased  that  remaining  budget  and  resulted  in  a 
small  over-run. 


Technical  Debt  Cost  (phase) 


Months  Since  Contract  Award 
|  —  ACWP  of  T echnical  Debt  —  Budget  Remaining  | 


Figure  13.  ACWP  of  technical  debt 


Task  2  -  Experience  Improvements 


As  the  EA  was  piloted  by  more  organizations  and  used  in  a  variety  of  modes,  it  became  clear  that 
there  were  a  number  of  shortfalls  in  the  user  interface,  the  underlying  simulation  models,  and  in 
the  scope  and  the  level  of  experience  throughout  the  life  cycle.  We  addressed  these  shortfalls 
by: 

•  Developing  a  richer  and  easier-to-navigate  user  interface 

•  Extending  the  experience's  scope  from  its  focus  on  preliminary  design  review  to  critical 
design  review  to  include  all  phases  through  low-rate  initial  production 

•  Tuning  the  simulation  to  allow  for  more  realism  and  easier  level  of  difficulty  adjustment 

•  Improving  and  extending  the  mentor  and  help  functions,  and 

•  Updating  and  expanding  the  lecture  and  student  materials  associated  with  the  EA  and  the 
UAV  experience. 
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Development  of  Full  Lifecycle 


The  UAV  experience  was  originally  conceived  and  designed  to  encompass  the  design  and 
development  of  a  UAV  system  from  shortly  after  preliminary  design  review  through  low-rate 
initial  production.  An  initial  prototype  addressing  this  lifecycle  was  developed.  However,  the 
sponsor  requested  focused  subsequent  work  on  the  part  of  the  experience  from  shortly  after 
PDR  up  until  critical  design  review.  Thus,  a  major  focus  of  this  project  was  to  leverage  the 
innovations  produced  for  the  PDR-to-CDR  portion  of  the  experience  to  improve  and  complete 
the  remaining  phases.  Two  main  areas  needed  to  be  addressed:  extension  of  the  underlying 
systems  dynamics  simulation  models,  and  providing  the  additional  phases  with  a  commensurate 
level  of  detail  in  the  user  experience. 


Systems  Dynamics  Simulation  models 

Changes  to  the  underlying  systems  dynamics  simulation  models  were  critical  to  extending  the  life 
cycle  as  well  as  improving  the  user  experience.  As  discussed  under  Task  1,  the  concept  of 
technical  debt  between  phases  was  one  critical  improvement  to  enable  more  realistic  transitions 
between  phases.  However,  additional  adjustments  to  the  existing  models  were  also  needed. 

There  are  different  models  applying  to  the  lifecycle  as  follows: 

•  Phase  2  -  PDR-to-CDR.  This  is  the  experience  phase  that  was  fully  developed  previously. 

•  Phase  3  -  CDR  to  Flight  Readiness  Review.  In  this  phase,  the  different  sub-systems  are 
integrated  into  test  articles  that  will  be  flight-tested. 

•  Phase  4  -  Flight  Readiness  Review  to  Production  Readiness  Review.  In  this  phase,  the 
UAV  system  is  transition  from  development  to  initial  production. 

•  Phase  5  -  Production  Readiness  Review  to  In  Service  Review.  In  this  phase,  the  UAV 
system  is  initially  produced  in  terms  of  ground  stations  and  vehicles. 

These  models  have  shared  variables  that  ensure  certain  values  are  kept  consistent  throughout 
the  experience.  The  following  work  was  performed  to  make  the  Phase  3,  Phase  4,  and  Phase  5 
simulation  models  consistent  with  the  improvements  previously  made  to  the  Phase  2  model. 

•  The  dates  for  the  various  reviews  (CDR,  FRR,  PRR,  ISR)  were  changed  into  variables  that 
can  be  changed  by  the  learner  to  reflect  either  schedule  advancements  or  delays  of  these 
reviews. 

•  Previously,  the  learner  set  staff  levels  as  one  of  the  recommendations  for  each  sub¬ 
contractor.  This  allowed  management  of  schedule  and  budget.  To  reflect  the  situation 
that  staff  levels  typically  do  not  adjust  automatically  but  rather  ramp  up  over  time,  the 
learner  decisions  were  changed  to  staff  level  targets.  The  simulation  then  moves  staffing 
levels  to  these  targets  over  time.  These  targets  are  for  experienced  personnel  and 
inexperience  personnel  in  each  sub-contractor,  as  well  as  the  system  integrator. 
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•  The  technical  performance  model  was  updated  to  reflect  the  addition  of  loiter,  true 
airspeed  and  payload  weight  to  the  range  calculation  for  the  aerial  vehicle.  Loiter  is  the 
time  that  the  UAV  can  hover  over  a  target  for  mission-related  purposes.  In  addition,  drag 
and  lift  were  changed  to  drag  coefficient  and  lift  coefficient.  Propulsion  efficiency  was 
changed  to  thrust  specific  fuel  consumption  (TSFC)  to  be  consistent  with  a  turbofan  jet. 
Figure  14  shows  the  drag  coefficient  across  Phase  2  and  Phase  3,  illustrating  also  the 
notion  of  shared  variables  across  phases. 


TPM:  Drag  Coefficient  for  Airframe  &  Propulsion  (program  life) 
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Figure  14.  Drag  coefficient 

•  Drag  reduction  and  TSFC  improvement  programs  were  added  as  options  for  the  learner. 
The  learner  can  set  a  desired  target  for  both  these  items  to  improve  range.  If  the 
programs  are  enabled,  a  fixed  number  of  technical  staff  are  purposed  for  the  program.  A 
variable  number  of  staff  are  assigned  depending  on  the  deviation  of  the  actual  drag  or 
TSFC  from  the  desired  targets.  Having  staff  allocated  to  these  programs  makes  them 
unavailable  for  work  aimed  at  meeting  the  entrance  criteria  for  the  next  review  meeting, 
thus  affecting  schedule  and  budget  in  the  earned  value  management  (EVM)  calculations. 
Figure  15  illustrates  the  revised  technical  performance  model  that  reflects  the  drag  and 
propulsion  efficiency  improvement  programs  (sub-model  on  left). 
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Figure  15.  Technical  performance  model 


•  Similarly,  a  weight  reduction  program  was  enabled.  This  exists  continuously  in  the 
experience,  since  aerial  vehicle  weight  problems  are  a  main  driver  of  the  poor  range 
performance.  The  learner  can  shift  weight  allocations  between  the  two  vehicle  sub¬ 
systems  (airframe  &  propulsion  versus  command  &  control).  For  a  particular  sub-system, 
the  model  allocates  technical  staff  to  weight  reduction  if  the  actual  weight  is  near  the 
weight  allocation.  If  the  actual  weight  is  over  the  weight  allocation,  then  the  model 
allocates  significantly  more  technical  staff  to  weight  reduction.  The  weight  reduction 
effect  is  a  function  of  the  allocation.  The  additional  staff  allocated  to  weight  reduction 
takes  away  staff  that  work  on  entrance  criteria.  Thus,  there  is  an  effect  on  schedule  and 
budget  for  poor  weight  allocation  decisions.  Figure  15  also  shows  the  weight  reduction 
program  model  (sub-model  on  right) 

•  Burn  rates  were  added  to  reflect  monthly  spending  by  each  of  the  sub-contractors,  the 
system  integrator,  and  the  overall  program. 

•  Productivity  rates  were  added  so  that  the  learner  can  see  changes  over  time. 

•  Charts  were  added  reflecting  the  number  of  experienced  vs.  inexperience  technical  staff 
currently  working  for  each  sub-contractor  and  the  system  integrator  (in  terms  of  full-time 
equivalents). 

•  Software  error  rates  were  adjusted  so  that  total  errors  encountered  and  total  errors 
resolved  are  tracked.  Figure  16  shows  a  graph  tracing  critical  errors. 


Report  No.  SERC-2017-TR-167 


23 


August  8,  2017 


Quality:  Critical  Software  Errors  in  Command  &  Control  System  (program  life) 


Months  Since  Contract  Award 

|  —  Errors  Outstanding  — Historical  Error  Target  — Errors  Resolved  — Total  Errors  — Historical  Error  Target  | 


Figure  16.  Critical  software  errors 

•  The  reliability  KPP  and  thermal  management  tracking  were  added  to  the  subsequent 
phases. 


Depth  of  User  Experience 

Extending  the  detailed  interactions  that  evolved  in  the  PDR  to  CDR  phase  to  the  remaining  phases 
was  more  difficult  than  envisioned.  Some  parts  were  relatively  easy.  For  example,  the  dashboard 
concept  could  be  easily  extended  to  include  the  data  items  that  were  used  in  Phase  2  as  well  as 
the  relatively  fewer  phase-specific  values  to  be  tracked  (e.g.  unit  cost,  completion  of 
documentation  required  for  the  end-of-phase  review). 

In  contrast,  when  it  came  to  the  story  line  for  the  remaining  phases,  the  original  prototype  design 
document  was  less  detailed  in  its  descriptions.  The  most  frequent  comment  on  decisions  and 
activities  was  "Same  as  Phase  2."  The  information  describing  the  phases  primarily  resided  in 
tables  of  decisions,  actions  and  expected  results;  little  was  provided  on  the  specific  information 
that  should  lead  the  learner  to  make  the  decisions  or  take  the  actions.  Additionally,  the 
simulation  models  needed  to  be  created  in  order  to  provide  the  data-driven  responses,  models 
were  not  created  until  late  in  the  project,  and  some  of  the  expected  variables  were  not 
implemented.  The  level  of  effort  required  to  match  the  Phase  2  depth  of  experience  in  Phases  3, 
4  and  5,  was  unexpected,  leaving  that  goal  only  partially  achieved.  In  the  end,  the  released 
version  used  simple  dialogs  from  the  original  prototype  with  slight  adjustments.  Evolving  the 
depth  of  the  later  phases  will  continue  to  be  refined. 
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Simulation  Tuning 


The  simulation  models  were  tuned  to  help  ensure  consistency  across  the  different  phases.  This 
work  entailed  the  following: 

•  Adjusting  cost  factors  in  the  EVM  modules 

•  Adjusting  staff  levels  among  the  different  sub-contractors 

•  Adjusting  hiring  and  attrition  rates  of  technical  staff 

•  Adjusting  factors  related  to  the  reliability  KPP  and  thermal  management  modules 


Role  of  NPC  Mentor 

Expanding  the  scope  of  Chris  Wilson's  role  as  mentor  provided  an  opportunity  to  integrate  the 
mentor,  the  library  and  the  help  system  to  better  support  the  learner. 

Originally,  the  mentor  was  intended  to  provide  help  by  explaining  some  of  the  technical  and 
programmatic  issues.  Feedback  from  instructors  and  students  suggested  expanding  the  role  to 
cover  additional  information,  including  how  to  actually  navigate  the  experience  and  what  to  do 
when  a  learner  was  "stuck." 

The  team  considered  several  approaches  to  meeting  the  expressed  feedback  and  implemented 
the  following  features,  dividing  the  information  provided  between  the  mentor,  the  help  system, 
and  the  home  page. 

•  We  established  the  "home"  page  as  a  place  where  instructions  to  the  learner  may  be 
provided  as  a  preview  (orientation)  for  each  phase.  Cues  as  to  what  types  of  things  the 
learner  might  expect  to  see  or  need  to  know  can  be  included.  While  the  experience  has 
default  contents,  it  can  be  edited  or  replaced  by  the  instructor  to  fit  the  class. 

•  We  expanded  the  dialogs  with  Chris  to  address  additional  subjects. 

•  We  created  an  initial  Help/FAQ  page  to  providebb  some  of  the  information  to  the  learner 
directly  and  in  a  more  familiar  way;  the  FAQs  will  be  augmented  with  additional  questions 
captured  as  more  learners  and  instructors  use  the  EA  and  provide  the  information  to  the 
developer/maintainer. 

•  The  FITML  implementation  makes  it  easier  to  linkto  outside  resources  from  the  dialogues, 
library  and  FAQs.  Some  of  the  links  provided  links  are  to  DoD  guidebooks,  the  DOD  EVM 
webpage,  the  INCOSE  SE  Body  of  Knowledge  wiki,  several  useful  blog  posts  and  "Ask  a 
Professor"  articles. 

•  We  created  a  more  active  role  for  the  mentor  to  literally  guide  the  learner  through  the 
trade  study.  Chris  acts  as  an  interface  to  outside  "experts"  that  provide  information, 
insight  and  artifacts  to  help  conduct  the  study. 
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Update  to  Lecture  and  Student  Materials 


The  lecture  and  student  materials  have  been  updated  during  this  development  cycle.  The 
instructors'  guide  has  been  updated  to  version  2.6  with  new  simulation  charts  outputs  and 
refreshed  content  of  the  recommendation  form.  Since  the  Experience  Accelerator  UAV 
experience  has  been  extended  to  representing  the  entire  acquisition  cycle,  new  information 
about  Phase  3-5  is  now  available.  The  instructor  and  student  materials  have  been  updated  to 
include  this  information.  A  section  describing  the  CDR  Criteria  functionality  that  can  be  used  to 
tune  the  difficulties  of  the  experience  was  also  added  to  the  .instructor  materials 


Task  3  -  Section  508  Compliance 


In  1998,  Congress  amended  the  Rehabilitation  Act  of  1973  to  require  Federal  agencies  to  make 
their  electronic  and  information  technology  (EIT)  accessible  to  people  with  disabilities.  The  law 
(29  U.S.C.  §  794  (d))  applies  to  all  Federal  agencies  when  they  develop,  procure,  maintain,  or  use 
electronic  and  information  technology.  Under  Section  508,  agencies  must  give  disabled 
employees  and  members  of  the  public  access  to  information  that  is  comparable  to  access 
available  to  others. 

Implementation  of  Section  508  also  amends  the  Federal  Acquisition  Regulation  (FAR)  to  ensure 
that  agency  acquisitions  of  EIT  comply  with  the  Access  Board's  standards.  The  FAR  change 
implementing  Section  508  was  published  along  with  an  explanatory  preamble  in  the  Federal 
Register  on  April  25,  2001  (66  Fed.  Reg.  20894)  and  is  effective  as  of  June  25,  2001.  The  FAR  rule 
can  be  found  on  the  Section  508  website. 

Section  1194.24(c)  and  (d)  of  the  Access  Board's  standards  require  that  all  training  or 
informational  video  and  multimedia  productions  which  support  the  agency's  mission  and  which 
have  audio  information  or  visual  information  that  is  necessary  for  the  comprehension  of  the 
content,  be  captioned  or  audio  described.  Flence,  if  the  production  is  multimedia  (e.g.  image  and 
sound)  and  is  considered  "training  or  informational,"  then  it  must  meet  the  applicable 
requirements  of  1194.24  (c)  and  (d)  of  the  Access  Board's  standards.  If  the  production  is  web- 
based,  regardless  of  whether  it  is  multimedia,  such  as  a  live  webcast  of  a  speech,  then  it  must 
also  meet  the  applicable  requirements  of  1194.22.  For  this  reason,  we  assert  that  the  SEEA  is  an 
education  tool  with  a  web-based  user  interface,  and  acknowledge  Section  508  applies  to  its  user 
interface  design  and  construction. 

Outside  of  the  US  federal  government,  a  number  of  voluntary  consensus  standards  have  been 
developed  by  standards  organizations  worldwide  over  the  past  decade.  Examples  of  these 
standards  include:  the  Web  Accessibility  Initiative's  Web  Content  Accessibility  Guidelines  (WCAG) 
2.0,  EN  301  549  VI. 1.1  (2014-02),  "Accessibility  requirements  for  public  procurement  of  ICT 
products  and  services  in  Europe,"  and  the  Fluman  Factors  Ergonomics  Society's  ANSI/FIFES  200.2 
(2008)  ergonomics  specifications  for  the  design  of  accessible  software. 
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This  evolution  of  these  global  standards  has  led  the  Access  Board  to  propose  revisions  to  the 
existing  508  Standards  to  harmonize  it  with  the  newcomers.  They  believe  such  a  harmonization 
would  support  the  goals  of  the  US  law,  while  also  creating  a  larger  marketplace  with  commercial 
competition  that  could  lower  costs  to  federal  agencies  as  well  as  commercial  manufacturers  for 
providing  accessible  electronic  information  and  communication  technology. 

Although  the  changes  may  take  some  time  to  be  developed,  reviewed  and  released,  we  intend 
to  comply  with  the  current  Section  508  requirements  and  include  as  many  additional 
requirements  from  the  industry  standards  as  seem  appropriate.  This  section  describes  the 
research  that  was  conducted  to  support  this  goal  in  the  following  major  activity  areas: 

•  T3.1:  Determine  methods  for  evaluating  Section  508  compliance,  authoring  content  and 
developing  simulated  learning  applications 

•  T3.2:  Determine  and  implement  necessary  changes  for  Section  508  compliance 

•  T3.3:  Translation  of  EA  screens  (login,  multi-player,  desktop,  dashboard,  etc.)  to  compliant 
formats 

•  T3.4:  Translation  of  EA  artifacts  (documents,  email,  recommendation  forms,  charts, 
dialog)  to  compliant  formats 

•  T3. 5:  Evaluation  of  508  compliance 

For  more  information,  see  the  document  "RT  167:  Certifying  SEEA  Compliance  with  Section  508", 
Technical  Report  No.  SERC-2016-TR-116,  (Task3,  Subtasks  3. 1-3. 2),  November  1,  2016. 


Methods  for  Evaluating  Section  508  Compliance 

We  took  a  constructional  approach  to  Section  508  compliance.  As  we  moved  forward  with 
building  out  the  initial  UAS  experience,  we  adapted  the  existing  software  for  Section  508 
compliance.  The  following  sections  describe  the  three  ways  in  which  we  accomplished  this. 


Flash  to  HTML  Conversion 

The  EA  User  Interface  was  originally  developed  using  Adobe  Flash  for  displaying  information.  This 
particular  technology  has  been  overtaken  by  the  capabilities  include  in  newer  web-based 
standards,  primarily  FITML5.  The  flash  technology  was  neither  designed  for,  nor  easily  adapted 
in  order  to  meet  Section  508  requirements.  Flowever,  early  in  its  evolution,  FITML  acknowledged 
accessibility  as  a  significant  design  factor1,  and  included  a  growing  number  of  accessibility-based 
capabilities.  The  SEEA  team  leveraged  these  capabilities  by  converting  the  existing  flash  interface 
to  HTML5. 


1  “Good  semantic  HTML  also  improves  the  accessibility  of  web  documents  (see  also  Web  Content 
Accessibility  Guidelines).  For  example,  when  a  screen  reader  or  audio  browser  can  correctly  ascertain  the 
structure  of  a  document,  it  will  not  waste  the  visually  impaired  user's  time  by  reading  out  repeated  or 
irrelevant  information  when  it  has  been  marked  up  correctly.  “https://en.wikipedia.org/wiki/HTML. 
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This  conversion  also  provides  more  flexibility  and  functionality  for  current  and  future  experience 
developers  than  the  current  implementation.  In  particular,  it  allows  us  to  develop  a  generalized 
tool  with  tailorable  templates  to  support  designing  user  interface  displays  and  reactive  features 
for  new  and  existing  experiences  that  align  with  their  specific  environments  and  learning 
objectives. 


Compliance  through  templates 

Several  HTML5  web  page  templates  have  been  developed  in  the  past  few  years  that  are  already 
certified  to  meet  the  Section  508  requirements.  There  are  also  numerous  open  source  and 
freeware  accessibility  components  designed  to  work  with  HTML5.  By  redesigning  the  EA  interface 
and  adopting  the  templates,  much  of  the  compliance  design  burden  is  lifted.  The  templates  that 
we  have  identified  have  saved  significant  effort  by  providing  display  methods  that  are  by  default 
constrained  to  accessible  requirements  such  as  font,  color,  contrast,  and  interoperability  with 
common  screen  readers  and  other  accessibility  enhancement  tools.  Using  a  template  also  greatly 
reduces  the  work  required  to  build  the  user  interface  design  tool. 


Certification  by  tools  and  experts 

There  are  also  automated  compliance  checkers  that  evaluate  web  pages  using  HTML5  code  and 
other  languages  to  assure  compliance  to  a  variety  of  accessibility  standards  of  the  webpage 
software  as  well  as  browser-specific  concerns.  We  do  not,  however,  believe  that  running  such 
checkers  constitutes  a  sufficient  compliance  case.  We  have  identified  additional  compliance¬ 
checking  tasks  as  we  designed  and  developed  the  interfaces.  We  have  consulted  with  Section  508 
experts  at  DAU  to  be  the  final  authority  on  compliance  certification  with  DAU  testing  providing 
validation  of  the  SERC  compliance  testing.  In  addition,  we  had  a  Section  508  expert  from  the 
Georgia  Tech  Research  Institute  review  the  EA.  We  used  approaches  and  tools  recommended  by 
the  DAU  experts  to  test  the  HTML  pages  and  contents.  The  used  tools  are  JAWS  screen  reader 
and  the  Microsoft  Active  Accessibility  Object  Inspector.  We  also  utilized  Tab-check  and  link 
checks  to  ensure  all  the  links  and  contents  are  accessible. 


Required  Changes  to  EA  Infrastructure  for  Compliance 

A  number  of  changes  beyond  those  originally  identified  were  necessary  in  the  EA  Infrastructure 
to  support  the  transition  from  Flash  to  HTML5.  The  necessary  changes  involved  the  following 
functionality: 

•  User  Interface:  content  retargeted  from  Flash  to  HTML5.  This  work  was  anticipated  and 
planned. 

•  Dialog  Engine:  While  the  dialog  engine  continues  to  utilize  Extensible  Markup  Language 
(XML)  generated  from  ChatMapper,  the  specifics  of  the  new  HTML5  user  interface 
required  changes  to  a  JavaScript  Object  Notation  (JSON)  and  XML  based  dialog  parser.  A 
good  portion  of  this  work  was  not  anticipated. 
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•  Client-Server  Communication:  The  TPC  sockets-based  communication  protocol  used  in 
the  Flash-based  systems  is  not  compatible  with  HTML5.  Thus,  a  new  communication 
system  was  written  utilizing  Web  sockets.  While  this  involved  a  substantial  design 
implementation  effort,  the  net  result  is  a  client  application  that  does  not  require  the 
opening  of  special  ports  as  was  required  for  the  Flash  implementation.  Thus,  the  FITML5 
implementation  can  work  passing  through  DoD  or  corporate  file  walls  within  any  IT 
intervention,  thus,  enabling  the  support  of  the  EA  through  a  standard  web-based  third- 
party  server.  This  required  a  substantial  amount  of  additional  implementation  effort. 


Translation  of  SEEA  Screens  to  Compliant  Formats 

The  existing  user  interface  uses  Adobe  Flash  technology.  Flash  technology  does  not  provide  the 
essential  components  for  Section  508  compliance.  The  updated  interface  includes  translation  and 
consolidation  of  the  existing  pages,  and  the  updated  pages  utilize  Section  508  compliant  assets. 
Table  1  depicts  the  changes  made  to  the  current  user  interface  pages.  The  entries  in  the  508 
Issues  column  refer  to  the  Section  508  paragraphs.  These  are  described  in  Appendix  A. 


Table  1  -  Changed  User  Interface  Pages 


Page 

Subpage 

Description 

508  Issues 

User Profile Design 

Page  displaying  user's  profile 

(a)(1) 

Profile Update Design 

Profile  update  page 

(n) 

Profile Design 

Welcoming  screen 

Logout Design 

Logout  screen 

Login Design 

Login  screen 

(n) 

Error Design 

Error  screen  overlay 

(1) 

File Explorer Design 

File  explorer  screen 

(1) 

ProjectStatus Folder Design 

Chart  viewer  screen 

(a)(e)(f)(i)(l) 

Email Design 

Email  screen 

(1) 

lnbox Subpage Design 

Inbox  subpage 

(1) 

Compose Subpage Design 

Compose  subpage 

(1) 

Desktop Design 

Desktop  screen 

(1) 

RecommendationForm report Design 

Recommendation  form  page 

(l)(n) 

Feedback Form Design 

Feedback  form  page 

(k) 

Multiplayer Main Design 

Multiplayer  main  page 

(1) 

Multiplayer Waiting Design 

Waiting  page  for  multiplayer 

(1) 

Multiplayer_Lobby_Design 

Game  searching  page  for 
multiplayer 

(n) 

Multiplayer Join Design 

Join  page  for  multiplayer 

(n) 

Multiplayer_Create_Design 

Game  creation  page  for 
multiplayer 

(n) 

Dashboard Design 

Dashboard  page 

(e)(f) 

Callv2_Design 

Phone  call  main  page 

(e)(1) 

Report  No.  SERC-2017-TR-167 


29 


August  8,  2017 


Call Subpagev3 Design 

Phone  call  subpage 

(e)(1) 

Chat Subpage Design 

Web  chat  subpage 

(e)(1) 

Voicemail_Subpagev2_Design 

Voicemail  subpage 

(e)(1) 

Translation  of  SEEA  Artifacts  to  Compliant  Formats 

The  EA  system  utilizes  document  artifacts  in  PDF/SWF  format  which  is  not  Section  508  compliant. 
To  achieve  compliance,  we  used  Adobe  or  third-party  software  to  make  the  necessary 
conversions.  The  artifacts  that  required  updates  were: 

•  DUAVCdrFormp2.swf 

•  DUAVCdrFormp3.swf 

•  DUAVCdrFormp5.swf 

•  Pl-UAV-Program-Info.swf 

•  P3-UAV-Program-Info.swf 

•  P4-UAV-Program-Info.swf 

•  P5-UAV-Program-Info.swf 

•  UAV  Background.swf 

•  UAV-CDR.swf 

•  UAV-PDR.swf 

•  UAV-Status-Chart-Info.swf 

•  UAV-Status-Chart-Info-0.21.swf 

•  EVM-Gold-Card-Info.swf 

•  IUISpl.swf 

•  IUISp2.swf 

•  IUISp3.swf 

•  IUISp4.swf 

•  IUISp5.swf 


Evaluation  of  Section  508  Compliance 

An  external  expert  on  Section  508  compliance  from  Georgia  Tech  Research  Institute  (GTRI) 
reviewed  the  application  and  code  and  discovered  several  accessibility  issues  which  are  described 
below  in  this  report.  All  of  the  issues  noted  have  addressed  in  the  final  version  of  the  code 
delivered  by  this  project. 
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Accessibility  Evaluation 


Section  508  of  the  Rehabilitation  Act  was  recently  amended  to  reference  the  W3C  Web  Content 
Accessibility  Guidelines  (WCAG)  2.0  and  apply  Level  A  and  Level  AA  Success  Criteria  and 
Conformance  Requirements  to  websites  as  well  as  non-web  electronic  documents  and  software. 
Compliance  with  the  updated  Section  508  standards  is  effective  January  18,  2018.  To  ensure 
future  compliance  with  accessibility  legislation  and  best  practices,  the  application  was  evaluated 
against  the  new  Section  508  criteria.  The  following  is  a  list  of  issues  found  when  inspecting  the 
application  for  compliance  with  Section  508  and  WCAG  2.0. 


Document  Language 

Each  page  of  the  application  is  missing  the  <html  lang>  attribute.  Although  the  application  works 
with  assistive  technology,  WCAG  2.0  does  require  that  the  language  of  the  page  be  identified. 


Forms 

Form  controls  are  not  properly  associated  with  text  labels,  so  the  labels  may  not  be  presented  to 
some  screen  reader  users.  Looking  at  the  code,  it  appears  that  the  ngAria  attribute  should  be 
included  to  programmatically  associate  form  controls  with  their  labels. 

Another  issue  seen  on  the  Recommendations  form  is  that  there  is  no  fieldset  provided  with  the 
Schedule  Confidence  Level  radio  buttons.  These  radio  buttons  should  be  marked  up  as  belonging 
to  a  fieldset  and  the  Schedule  Confidence  Level  label  should  be  associated  as  the  description  for 
that  fieldset. 


Color  Contrast 

One  of  the  WCAG  2.0  success  criteria  that  will  be  new  to  Section  508  is  the  requirement  for 
minimum  levels  of  color  contrast.  In  general,  the  color  contrast  meets  the  Section  508  and  WCAG 
2.0  requirements.  However,  there  are  a  few  exceptions. 

On  the  Login,  FAQ,  Dashboard,  and  Graph  Viewer  pages,  the  header  at  the  top  of  the  page  ("SERC 
Systems  Engineering  Experience  Accelerator,"  "FAQ,"  "Dashboard,"  and  "Graph  Viewer")  fails  the 
WCAG  2.0  requirements  with  a  contrast  ratio  of  only  1.4:1.  The  text  color  should  be  much  darker 
to  improve  visibility. 

On  the  Home  page,  the  Help  and  Reset  Experience  buttons  fail  the  WCAG  2.0  requirements  at 
the  AA  level.  The  buttons  either  need  to  be  darker  or  the  text  color  changed  to  black.  The  Logout 
button  on  the  Home  page  as  well  as  the  Disconnect  button  on  the  Phone  page  also  fail  the 
requirements  at  the  AA  level  for  normal  text  size  but  passes  for  large  text.  Increasing  the  text 
size,  making  the  red  background  darker,  or  changing  the  text  color  to  black  would  alleviate  these 
contrast  issues. 
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On  the  Recommendation  page,  the  labels  for  the  Schedule  Confidence  Level  Low  and  Moderate 
radio  buttons  are  low  contrast.  The  text  needs  to  be  darker  to  pass  the  WCAG  2.0  Level  AA 
requirements. 


High  Contrast  Color  Scheme 

All  of  the  content  remains  visible  and  accessible  when  a  high  contrast  color  scheme  is  activated. 
High  contrast  color  schemes  are  useful  for  some  individuals  with  low  vision. 


Disabled  Style  Sheets 

When  style  sheets  are  disabled,  a  number  of  links  all  titled  Recommendation  Form  appear  with 
the  navigation  links.  This  can  be  confusing  especially  since  there  is  no  discernable  difference 
between  all  of  the  links. 


Task  4  -  Validation  of  Research  Hypothesis  via  Learning  Evaluation 


The  EA  provides  a  simulation-based  systems  engineering  project  experience  which  is  coupled 
with  learning  assessment  capabilities.  During  this  research,  significant  work  have  been 
completed  to  improve  the  capabilities  of  the  EA,  namely  the  improvements  on  the  simulation 
experience,  the  learning  assessment  capabilities  and  the  toolset  for  analyzing  learning  data.  In 
addition,  efforts  have  been  made  to  expand  the  use  of  the  EA  into  a  number  of  different  domains 
and  learning  environments. 


Support  for  DAU  and  Other  Institutions 

During  the  development  period  of  this  project,  numerous  institutions  sought  the  usage  of  the 
Experience  Accelerator  and  the  team  has  ensured  that  these  deployments  were  well  supported. 
During  this  development  cycle,  multiple  pilot  uses  were  conducted.  Instructors  from  University 
of  Alabama  in  Huntsville  (UAH)  and  Airforce  Institute  of  Technology  (AFIT)  showed  interest  of 
using  Experience  Accelerator  in  their  Systems  Engineering  courses.  As  a  result,  there  have  been 
four  pilot  uses  conducted  in  UAH  so  far,  and  new  pilot  uses  of  EA  in  UAH  and  AFIT  schedule  for 
the  fall  of  2017. 


Pilot  UsesofSEEA 

The  following  is  a  list  of  the  pilot  uses  of  the  SEEA. 

Pilot  use  of  the  EA  with  multiplayer  mode:  UAH  Feb  2016 

The  EA  was  used  in  a  UAH  course  in  multiplayer  mode,  and  performance  data  was  gathered  for 
learning  evaluation. 
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Pilot  use  of  the  EA  with  single  player  mode:  UAH  Nov  2016 

The  EA  was  used  in  a  UAH  course  in  single  player  mode,  performance  data  was  gathered  for 
learning  evaluation. 

Pilot  use  of  the  EA  with  single  player  and  multiplayer  mode:  UAH  Feb.  2017  -  May  2017 

The  EA  was  used  in  a  UAH  spring  course  in  both  single  and  multi-  player  mode.  Performance  data 
was  gathered  for  learning  evaluation. 

Pilot  use  of  the  EA  with  single  player  mode:  DAU  June  2017 

This  pilot  will  be  conducted  with  DAU  teaching  faculty  to  review  its  use  for  the  DAU.  Feedback 
was  gathered  on  the  format  and  content  of  the  DAU  experience. 

Pilot  use  of  the  EA  with  single  player  mode:  UAH  Aug  2017 

This  pilot  will  be  conducted  targeted  to  begin  on  Aug  16th.  Multiple  runs  and  control  group  data 
will  be  gathered. 

Pilot  use  of  the  EA  with  single  player  mode:  AFIT  Aug  2017 

This  pilot  will  be  conducted  targeted  to  begin  on  Aug  18th.  Performance  data  of  systems 
engineering  experts  will  be  gathered  during  this  pilot. 


Learning  Evaluation  Design  and  Data  Collection 

Learning  assessment  is  a  critical  component  of  accelerated  learning.  It  is  crucial  to  understand 
the  learning  results  and  the  efficacy  of  different  types  of  learning  experiences.  This  is  imperative 
both  in  assessing  the  capabilities  of  the  learner,  but  also  to  improve  the  efficacy  and  the 
capabilities  of  the  learning  experience.  While  assessment  capabilities  are  critically  important, 
nothing  was  found  in  the  literature  that  was  directly  applicable  to  automated  assessment  of 
systems  engineering  skills  in  the  SEEA.  Therefore,  a  new  experimental  design  grounded  in  the 
literature  was  devised,  along  with  a  set  of  tools  to  facilitate  its  application. 

While  the  Experience  Accelerator  (EA)  has  a  broader  goal  of  accelerating  the  learning  of  critical 
SE  competencies  through  an  experience-based  system,  systems  thinking  skills  are  a  key 
component  of  the  targeted  learning  outcomes.  Systems  thinking  is  at  the  core  of  the  targeted  EA 
SE  competencies  and  therefore  one  of  the  primary  competencies  to  be  assessed  in  order  to 
evaluate  the  effectiveness  of  the  EA. 

Systems  thinking  seeks  to  improve  decision  making  and  complex  problem  solving  through  deep 
systemic  understanding.  Typically,  in  order  to  assess  learning  gains  in  these  areas,  three 
approaches  are  utilized:  measuring  performance  resulting  from  decisions  (such  as  a  game  or 
simulation  score),  reviewing  decisions  and  actions  that  were  taken,  or  measuring  learner 
understanding  (the  rules  and  mental  operations  that  lead  to  decision  making).  Measuring 
understanding  seeks  to  verify  that  improved  decision  making  arises  from  understanding  the 
system  and  not  simply  from  trial-and-error.  All  of  these  approaches  are  valid  and  can  result  in 
useful  evaluations.  As  systems  thinking  skills  are  applied  in  order  to  understand  and  solve 
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complex  problems,  educational  research  on  the  assessment  of  problem-solving  skills  can  be 
helpful  in  designing  an  effective  evaluation. 

In  order  to  solve  an  ill  structured  problem,  students  must  be  able  to  deconstruct  the  problem 
into  its  constituent  parts  (e.g.,  stakeholders,  relationships  among  them,  impacts  of  the  problem 
on  them),  define  the  problem  in  their  own  words,  determine  resources  to  help  them  understand 
the  problem,  determine  and  pursue  learning  issues,  and  develop  and  test  a  solution.  Research 
on  the  evaluation  of  problem-solving  skills  tells  us  that  in  order  to  evaluate  problem-solving 
ability,  we  must  assess  students'  ability  to  do  each  of  these  steps.  The  EA  seeks  to  accelerate  the 
learning  of  novice  SE's  and  advance  them  more  quickly  to  expert  SE  performance.  Experts  use 
heuristics  to  skip  steps;  novices  typically  are  not  capable  of  doing  this. 

A  meta-analysis  of  problem  solving  assessment  literature  found  that  18  of  23  studies  deemed  of 
high  quality  used  cases  or  simulations  as  assessment  methods.  With  the  EA  Simulation,  it  is 
possible  to  measure  learner's  performance  within  the  experience.  Learners  make  decisions 
within  the  EA,  the  simulation  determines  the  results  of  those  decisions,  and  we  are  provided  with 
outcomes  that  we  can  utilize  in  order  to  assess  the  effectiveness  of  learners'  decisions. 

In  order  to  assess  learners'  levels  of  understanding  and  to  determine  if  the  EA  improves  learning, 
a  more  thorough  picture  of  the  thinking  behind  learners'  choices  is  needed.  Therefore,  to  assess 
learners'  understanding,  it  is  required  to  also  elicit  their  views  of  the  system,  the  problems  they 
faced,  and  the  thinking  behind  their  decisions  made  to  solve  these  problems.  Emerging  literature 
in  systems  dynamics  increasingly  has  been  seeking  to  assess  learners'  understanding  or  mental 
models. 

Therefore,  in  order  to  assess  learner  performance,  one  can  capture  it  through  EA  results,  and 
analyze  the  actions  and  decisions  taken  by  the  learner.  To  assess  learner  understanding,  learner 
approaches  to  decision-making  (through  verbal  protocols)  can  be  captured  and  expert  choices 
and  protocols  can  be  used  as  a  baseline  for  "good"  decision  making. 

The  evaluation  plan  therefore  focused  on: 

1.  Benchmarking  with  an  objective  "score"  which  is  also  useful  in  motivating  students. 

2.  Comparing  subject  matter  experts'  EA  actions  and  results  to  novice  systems 
engineers'  actions  and  results. 

3.  Comparing  subject  matter  experts'  written  (or  transcribed  verbal)  descriptions  of 
their  decision-making  process  during  the  EA  to  novice  systems  engineers'  written  (or 
transcribed  verbal)  descriptions  of  theirdecision-making  process  duringthe  EA  in  first 
and  second  run  through  the  SEEA  experience. 

4.  Tracking  learning  with  changes  in  1-3  above  through  a  learner's  multiple  iterations 
through  the  experience. 
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To  support  this  plan,  the  EA  was  instrumented  to  record  information  to  serve  as  a  learning 
laboratory.  Research  will  be  done  to  determine  the  requisite  data  that  needs  to  be  recorded  and 
the  EA  will  be  updated  accordingly.  Prior  to  completing  this  research,  the  following  data  has  been 
selected  and  will  be  collected  from  the  EA: 

•  Participant  Identification: 

o  Learner's  Name  &  demographic  information 
o  Team  Name  &  other  members 
o  Instruction  Name  &  Roles  played  in  Experience 

•  Experience  Session  information: 

o  Experience  Name  and  Version 
o  Date  of  Experience  Start  and  End 
o  login  dates  and  duration  of  each  session 
o  Phases/cycles  covered  in  each  login  session 
o  Elapsed  time  &  number  of  sessions  per  Phase/Cycle 
o  Links  to  past  experience  information 

•  Learner  Experience  Inputs  &  Actions: 

o  Self-Assessment 

o  Initial  Recommendation  Input 

o  All  subsequent  Recommendation  Inputs 

o  Workflow  sequence  with  each  action  recorded  with  a  timestamp 
o  Who  is  called  and  which  questions  are  asked,  in  which  order 

•  Instructor  Input 

o  Feedback  provided  to  Learners  (dialog,  email,  etc.) 
o  Recommendations  accepted/rejected 
o  Instructor's  observations 

•  Simulation  Output: 

o  Last  phase/cycle  completed 
o  Results  of  schedule,  cost,  range  and  quality 
o  Final  Status  Charts 
o  Final  score 

•  Reflection 

o  Reflection  feedback  provided  to  the  Learner 
o  Learner's  reflection  input 

Furthermore,  a  set  of  analysis  tools  are  being  developed  to  make  sense  of  this  information.  Test 
cases  are  being  created  to  provide  benchmarks  to  baseline  this  analysis.  The  tool  will  provide 
capabilities  to  compare  a  learner's  historic  experience  data  with  new  ones,  and  use  the  data  to 
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match  against  the  experts'  data  to  determine  the  maturity  levels  of  the  competencies.  Finally,  a 
demonstrable  set  of  learning  experiences  will  be  recorded  and  analyzed  to  provide  feedback  on 
the  capabilities  of  the  system. 


Data  Analysis  and  Evaluation  of  Learning  Effectiveness 

After  the  pilot  courses  were  completed,  the  performance  data  of  the  students  were  gathered 
and  compared.  The  performance  measures  include  range,  critical  software  defects,  schedule, 
CDR  artifact  completion  and  budget  overrun.  The  SEEA  combines  these  measures  to  determine 
if  the  CDR  can  be  achieved  successfully  and  determines  the  risk  to  proceed  with  the  UAV 
program.  During  the  pilot,  students  made  different  decisions  resulting  in  a  range  of  performances 
and  different  program  results.  Among  the  twenty-five  students  participating,  most  of  them  were 
able  to  complete  the  whole  project  cycle  and  reach  Phase  7  to  receive  performance  feedback 
from  the  SEEA. 

The  data  gathered  during  the  pilot  application  was  analyzed  to  provide  insights  on  the  students' 
decision  making,  their  capability  to  discover  issues  in  the  system,  their  ability  to  prioritize 
resources  and  the  outcomes  of  their  decisions.  As  mentioned  earlier,  many  different  types  of 
data  is  gathered  by  the  SEEA  system.  Participant  identification  and  experience  session 
information  are  used  to  identify  the  specific  learners  and  their  use  of  the  system.  Learner 
experience  inputs  and  actions  are  valuable  data  to  track  the  learner's  actions  and  behaviors 
during  the  experience,  which  provides  insights  into  the  learner's  decision  making  process. 
Simulation  output  data  was  used  to  determine  the  general  performance  for  the  learner  and  it 
also  demonstrates  the  outcomes  of  learner  decisions.  Instructor  input  and  reflections  can  be  used 
to  evaluate  the  efficacy  of  the  learning  and  to  improve  the  learning  experience. 

One  example  is  in  range  performance.  At  the  beginning  of  Phase  2,  a  weight  increase  in  the 
Command  &  Control  System  (CCS)  will  cause  the  range  to  drop  significantly.  At  this  point, 
although  the  range  is  still  within  acceptable  range,  the  trend  line  is  very  problematic.  Many 
students  expressed  their  concerns  over  the  range  trend  and  the  defects  trends.  A  common 
student  reaction  to  it  is  to  add  more  staff  to  the  Airframe  and  Propulsion  System  (APS)  and  CCS 
development  organizations,  both  to  improve  schedule  and  quality.  As  shown  in  Figure  17, 
Student  #2  and  Student  #10  performed  very  well  in  the  range  performance  area.  As  shown  in 
Figure  18,  both  of  them  added  staff  to  APS  systems  to  the  level  of  80  or  more  staff  which  enabled 
them  to  successfully  address  the  range  issues.  The  approach  also  helps  to  reduce  the  drag 
coefficient  as  shown  in  Figure  19. 
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Figure  17  Range  Performance 
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Figure  18  APS  System  Sr.  Staff 
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Figure  19  Drag  Coefficient  Performance 

For  the  learning  assessment,  the  group  of  students  from  the  UAH  pilot  received  their  final  grade 
for  the  course.  The  final  grade  was  calculated  based  on  students'  performance  on  multiple  home 
works  and  exams.  The  EA  practice  was  counted  as  5%  towards  the  final  grade,  and  thus  the  direct 
impact  of  the  EA  score  was  low.  Figure  20  shows  the  correlation  between  the  final  grade  and  EA 
grade  given  to  the  students  by  the  instructor  based  on  the  students'  presentations  and  comments 
for  the  Experience  Accelerator  homework.  The  x-axis  indicates  the  homework  grade  received  by 
the  students,  while  the  y-axis  indicates  their  final  grades  for  the  course.  As  shown  in  the  figure, 
the  EA  homework  grades  are  not  significant  as  a  predictor  of  final  grade. 
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Figure  20  EA  Grade  and  Final  Grade 

Figure  21  shows  the  correlation  between  the  EA  Final  Score  and  the  Final  Grade.  By  comparison, 
the  EA  Final  Score  provides  greater  significance  as  a  predictor  of  the  Final  Grade.  The  EA  Final 
Score  was  calculated  based  solely  on  the  learners'  performance  in  the  experience.  Four  aspects 
of  the  performance  were  counted,  including  Quality,  Schedule,  Range  and  Cost.  Each  of  these 
aspects  weights  25%  towards  the  EA  final  score. 

Flowever,  in  both  cases,  the  r2  is  too  small  to  consider  the  homework  grade  or  EA  score  as 
significant  predictors  of  the  final  grade.  Flowever,  it  has  to  be  noted  that  the  EA  pilot  run  was 
conducted  at  the  beginning  of  the  course  and  without  a  second  run,  it  is  difficult  to  accurately 
evaluate  learning. 
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Figure  21  EA  Final  Score  and  Course  Performance  Regression 

Also  worth  noting  is  that  during  the  2017  Spring  UAH  pilot  application,  about  30%  of  the  students 
volunteered  to  use  the  EA  through  multiple  runs.  The  data  gathered  from  the  runs  indicate  that 
a  majority  of  the  students  who  volunteered  for  the  multiple  EA  runs  improved  their  performance 
on  the  Range,  Drag,  and  Weight  aspects.  These  students  generally  added  staff  in  APS  and  CCS 
development  earlier  in  the  development  phase.  Most  of  the  students  performed  better  in  each 
subsequent  run  through  the  experience.  After  analyzing  the  students'  updated  rationale  for  the 
changed  behavior,  some  degree  of  learning  can  be  traced  to  the  students  describing  previous 
performances  and  the  potential  approaches  to  avoid  previously  discovered  issues.  Students  who 
went  through  the  EA  experience  with  multiple  runs  appear  to  perform  slightly  better  in  the  class, 
with  an  average  grade  of  87%  comparing  to  an  average  grade  of  85%  by  students  who  went 
through  EA  only  once  in  the  semester.  However,  this  result  is  not  statistically  significant.  Figure 
22  shows  the  final  grade  for  the  students,  while  yellow  indicating  the  students  who  performed 
multiple  runs  through  the  EA. 
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Figure  22  Students  Course  Final  Grade 

Figure  23  and  Figure  24  show  the  Drag  Coefficient  Performance  and  the  Range  Performance  with 
the  comparison  of  2nd  run  of  the  EA  against  the  1st  run  of  the  EA  experience. 
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Figure  23  Drag  Coefficient  Performance 

For  students  who  participated  in  multiple  runs,  the  improvements  of  the  performance  were 
significant.  This  result  indicated  that  student's  performance  improves  during  multiple  runs  as 
indicated  in  EA  scores  which  is  reinforced  by  their  self-assessments.  Flowever,  it  has  not  been 
validated  through  a  test  group  the  degree  to  which  the  EA  contributes  to  overall  student  learning. 
Additional  research  is  necessary  in  this  area. 
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Figure  24  UAV  Range  Performance 

Feedback  from  the  instructor  was  very  positive  as  he  indicated  that  "the  value  of  the  EA  is  that 
it  integrates  into  a  dynamic  simulation  several  of  the  key  concepts  covered  in  the  systems 
engineering  class,  including: 

•  Technical  performance  measurement  and  tracking, 

•  Margin  management, 

•  Earned  value  management, 

•  Risk  analysis, 

•  The  systems  life  cycle,  and 

•  Technical  reviews. 

As  such,  it  greatly  augments  the  lectures  and  homework  assignments  for  these  topics,  which 
tend  to  be  focused  on  only  one  of  these  interrelated  topics." 

Task  5:  Deployment  Support  at  DAU 


The  following  is  a  description  of  the  concrete  deliverables  so  that  the  EA  can  be  used  in  the  DAU 
on  a  long-term  basis.  This  includes  criteria,  requirements,  sustaining  support,  technical  details  of 
the  hosting  requirements,  and  software  configuration  items. 


Software  and  Hardware  Requirements 

The  following  describes  the  software  and  hardware  requirements  for  the  EA  Server,  Client  and 
tools. 
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EA  Server 


The  following  are  the  requirements  for  the  EA  Webserver  and  application  server: 

Software  Requirements 

•  Operating  System  requirement  for  the  SEEA  software  deployment  is  Ubuntu  16.04.2  X32 

•  Required  Software  Installation: 

o  Phpmyadmin  Network  Administration  Tool 
o  MySQL  Database,  version  14.14 
o  Apache  2  Web  Server,  version  2.4.18 
o  JavaJRE7 

Hardware  Requirements2 

Performance  analysis  of  the  Flash-based  version  of  the  EA  was  conducted  and  documented  in  the 
document  "Developing  the  Systems  Engineering  Experience  Accelerator  (SEEA)  Prototype  and 
Roadmap",  Final  Technical  Report  SERC-2013-TR-016-3,  December  31,  2013.  It  was  found  from 
this  work  that  the  minimum  application  memory  size  to  avoid  performance  slowdowns  is  given 
by: 

Min_Memory_Size  =  181  MB  base  +  126.4MB  *#  processes 

It  was  also  found  that  the  time  dependence  on  CPUs  for  completion  of  a  phase  was  given  by: 

time  —  0.4  minutes  x  concurrentSimulations  +  #  cores 

Finally,  the  breakdown  of  CPU  utilization  during  the  Phase-2  initialization  and  Phase-2 
simulations  is  approximately: 

•  20%  -  startup  costs  (load  and  pre-process  model  XML) 

•  5%  -  actual  simulation 

•  70%  -  rendering,  encoding,  and  writing  graph  files 

o  20%  reading  CSV  files 
o  10%  drawing  of  the  graph 
o  20%  rasterizing  the  graph  to  an  image  buffer 
o  20%  encoding  the  image  buffer  and  saving  as  jpeg 

Below  is  the  current  EA  server  configuration  which  successfully  supports  approximately  six 
concurrent  users. 

•  One  CPU  core 

•  512MB  RAM 

•  20GB  HDD  or  SSD 

•  100/1000M  Internet  connection 


2  Can  be  either  virtual  machine  on  a  cloud  hosting  solution  or  physical  server. 
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Based  on  the  prior  work,  these  concurrent  users  would  experience  phase  updates  latencies  of 
approximately  2.4  minutes  due  to  the  CPU  loading,  and  would  experience  additional  delays  due 
to  the  relative  lack  of  RAM  (512MB  available  vs.  ~940MB  for  optimal  performance).  However, 
performance  optimizations  have  been  made  since  this  performance  analysis  work  was  completed 
and  there  are  other  mitigating  factors. 

First,  this  performance  analysis  assumes  that  the  users  will  simultaneously  enter  the  end  of 
Phase-2  cycle  which  requires  the  rendering  of  a  large  number  of  status  charts.  In  actual  usage, 
users  will  not  simultaneously  enter  an  end  of  phase.  If  learners  spend  an  average  of  15  minutes 
per  phase,  this  represents  a  duty  cycle  of  approximately  13%  for  a  two  minute  phase  end 
execution  time.  Also,  a  work  queue  was  added  for  simulation  runs  queuing  function  has  been 
added  to  the  EA  which  was  estimated  to  reduce  average  wait  time  by  between  25  and  50%, 
depending  on  the  number  of  users.  The  communication  times  have  also  been  reduced  through 
the  use  of  files  to  reduce  inter-process  communication. 

It  is  believed,  but  not  yet  verified,  that  the  HTML5-based  implementation  will  provide  less  of  a 
burden  on  the  server  due  to  smaller  Websocket  package  size  and  less  frequent  server/client 
communication.  A  special  version  of  the  HTML5  implementation  with  online  chart  generation  can 
eliminate  the  needs  for  simulator  to  generate  charts,  which  will  significantly  reduce  both  I/O  time 
and  simulator  running  time  (70%  of  execution  time  is  due  to  rendering,  encoding,  and  writing 
status  graph  files).  Future  research  is  necessary  to  measure  the  actual  impact  of  these 
performance  optimizations. 


EA  Client 

The  following  are  the  requirements  for  the  EA  Client: 

Software  Requirements 

•  HTML5  Compatible  browsers  which  currently  include: 

o  Microsoft  Edge  15 
o  Google  Chrome  55 
o  Mozilla  Firefox  53 
o  Apple  Safari  10 

Minimal  Hardware  Requirements 

•  Any  device  with  internet  connectivity  that  can  support  the  acceptable  browsers.  On 
small  screens  the  appearance  may  be  compromised,  but  the  functionality  is  intact. 


EA  Tools 

The  following  are  the  requirements  for  the  EA  Tools  support: 

Software  Requirements 

•  Microsoft  Windows  7  SP1  Operating  System 
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•  .NET  Framework  4.6 


Minimal  Hardware  Requirements 

•  Intel  Core  i3  CPU 

•  1GB  RAM 

•  1GB  HDD  or  SSD 


Hosting  Solutions 

The  following  is  a  description  of  the  possible  EA  application  hosting  solutions.  Options  include 
commercial  hosting  (external  network),  internal  hosting  (internal  network)  and/or  support  on  a 
laptop  within  the  classroom  (classroom  network). 


Commercial  Hosting 

From  a  logistical  perspective,  a  commercial  hosting  solution  is  the  simplest  to  implement.  The 
Experience  Accelerator  is  currently  being  hosted  on  a  commercial  server  at  the  URL 
http://www.experience-accelerator.org/.  This  server  can  be  provisioned  to  provide  adequate  to 
customers  to  ensure  adequate  response  times  for  instantaneous  usage.  Students  could  access 
this  website  as  they  would  any  other  website  anywhere  there  is  internet  access.  This  solution 
also  allows  for  easy  access  to  updates  in  the  Experience  Accelerator  infrastructure  and 
experiences,  and  external  support  for  the  hosting  services. 

There  are  three  challenges  with  this  approach.  The  major  one  is  potential  firewall  issues.  To 
facilitate  satisfactory  client/server  performance,  port  1935  and  80  are  required  to  be  opened  to 
support  the  sockets-based  client/server  communication  protocols  in  the  Flash-based 
implementation.  This  is  generally  not  a  problem  in  open  environments,  but  can  be  an  issue  in 
secure  facilities.  While  it  is  not  technically  difficult  to  open  firewall  ports,  this  usually  requires 
approval  by  the  organization's  IT  group,  and  there  may  be  official  policies  prohibiting  this  action. 
Fortunately,  with  the  HTML5  implementation,  this  is  no  longer  necessary.  The  removal  of  this 
restriction  has  been  validated  with  operation  within  the  DAU  and  other  secure  environments. 

The  second  potential  issue  is  with  the  proprietary  nature  of  an  organization's  experience  and 
learner  database.  If  an  organization  wishes  to  protect  these  details  with  greater  levels  of  security 
than  are  available  by  a  commercial  application  host,  then  additional  security  will  need  to  be 
instituted  or  this  secure  information  will  need  to  be  obscured  by  the  users. 

The  third  issue  is  with  configuration  management.  There  are  many  different  ways  in  which  a 
particular  experience  can  be  configured  in  the  EA.  While  it  is  preferable  that  each  organization 
has  the  ability  to  tune  each  experience  to  their  liking,  allowing  this  on  a  commercial  host  will 
require  the  ability  to  support  multiple  versions  of  each  experience.  Supporting  this  will  require 
the  development  of  new  features  in  the  Experience  Accelerator.  Such  efforts  can  be  achieved  by 
creating  a  web  interface  where  different  experiences  can  be  accessible  to  users  with  certain 
credentials.  One  potential  technical  challenge  are  the  TCP  ports  used  by  the  web  application.  The 
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current  HTML5  implementation  utilizes  port  80  for  the  websocket  connection  to  avoid  any 
firewall  issues.  Multiple  EA  instances  cannot  share  the  same  TCP  port  for  server-client 
communication,  therefore  it  may  lead  to  potential  firewall  issues  for  certain  organizations. 
However,  mitigation  options  are  available.  A  gateway  server  linking  to  multiple  experience 
servers  will  allow  the  use  of  port  80  and  at  the  same  time  provide  the  users  with  a  single  easy  to 
use  interface  to  get  access  to  the  experiences. 


Internal  Hosting 

This  solution  removes  the  security  issues  noted  above  for  the  commercial  hosting  solution,  but  it 
introduces  a  number  of  additional  problems.  The  most  significant  of  these  issues  is  the  fact  that 
software  needs  to  be  installed  inside  the  learner's  firewall.  In  many  organizations,  this  requires 
approval  by  the  IT  department  that  may  entail  a  lengthy  certification  process.  The  EA  contains  a 
significant  amount  of  open  source  software  which  might  be  an  issue  in  some  organizations.  This 
approach  will  also  hinder  the  organization's  ability  to  have  updates  installed  in  a  timely  manner. 
Finally,  it  will  require  internal  support  by  the  hosting  organization. 


Classroom  Support 

There  are  a  number  of  possible  solutions  that  could  be  supported  by  an  instructor  in  a  classroom 
situation  in  which  a  commercial  or  internal  hosting  solution  is  not  possible. 

Instructor  Hot-Spot 

In  the  case  where  external  internet  access  is  not  available,  the  first  solution  is  one  in  which  the 
instructor  provides  a  hot  spot  for  the  students  to  a  commercial  hosting  site.  If  this  does  not 
violate  security  rules,  this  would  be  a  simple  solution  with  the  advantages/disadvantage  of  the 
commercial  hosting  solution,  but  with  the  requirement  for  the  instructor  to  provide  an  internet 
hot-spot.  There  might  also  be  some  bandwidth  limitations  if  all  learners  instantaneously  initiate 
status  chart  downloads.  However,  given  the  availability  of  the  HTML5-based  EA,  it  is  believed  that 
only  in  rare  occurrences  would  the  learner  not  have  access  to  the  internet  through  a  corporate 
firewall. 


Instructor  Hosting 

Another  option  is  a  hybrid-solution  solution  for  classroom  situations  in  which  there  is  no  access 
to  the  internet.  In  this  case,  the  instructor  has  the  EA  server  application  installed  on  his/her 
laptop  and  allows  the  students  to  connect  to  his  system  through  a  personal  hotspot  or  through 
hardwired  Ethernet.  The  instructor  could  download  and  install  this  software  suite  on  his/her 
computer  through  a  download  facility  that  the  Experience  Accelerator  team  provides.  This 
approach  would  provide  the  instructor  with  the  ability  to  tune  the  experience  as  he/she  sees  fit 
and  would  provide  what  would  likely  be  a  low-latency  network. 

There  are  a  number  of  issues  with  this  approach.  First,  there  may  still  be  security  issues  if  the 
laptops  used  by  the  students  in  the  classroom  are  connected  to  the  organization's  network.  The 
other  issue  is  that  the  instructor  is  now  responsible  for  supporting  the  EA  application.  There  are 
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likely  to  be  performance  issues  based  on  the  capabilities  of  the  instructor's  computer.  Finally, 
unless  the  instructor  has  the  capability  to  use  his/her  computer  as  a  Webserver,  the  students  will 
only  have  access  to  the  application  when  they  are  in  the  classroom  with  the  instructor's 
computer. 


Learner  Hosting 

The  final  option  is  where  both  the  client  and  server  application  are  supported  on  each  student's 
computer  in  which  case  there  is  no  need  for  networking  or  internet  access.  This  could  be  done 
through  an  install  of  the  application  through  a  thumbdrive  or  through  a  CD/DVD  drive.  In  each 
case,  the  students  would  be  provided  with  the  necessary  software.  This  option  is  simple  in  that 
each  learner  can  work  independently  without  reliance  on  anything  that  is  not  on  their  computer. 
However,  it  does  have  a  severe  limitation  in  that  multi-player  cannot  be  supported  in  this  mode. 
In  addition,  installing  software  on  a  work  computer  usually  requires  internal  IT  acceptance  and 
support,  which  may  be  more  difficult  that  having  the  application  installed  on  an  internal  server. 
It  is  likely  that  unsecure  classroom  computers  would  need  to  be  used  for  this  purpose.  Thus, 
interactions  between  students  and  the  instructor(s)  will  need  to  take  place  by  means  outside  of 
the  EA  application.  Also,  the  instructor  will  not  have  direct  access  to  the  students'  experience 
data. 


Summary 

The  hosting  options  are  summarized  in  Table  2  shown  below.  Based  on  the  aforementioned 
advantages  and  disadvantages,  the  preferred  option  for  the  DAU,  and  most  other  organizations, 
is  to  use  the  third  party  commercial  approach.  However,  the  adopted  solution  depends  upon  the 
organization's  security  requirements  and  the  relative  value  and  cost  of  the  parameters  noted 
below. 


Table  2  -  Experience  Accelerator  Hosting  Option 


Approach 

Security 

Requirements 

Cost 

Support  Issues 

Capabilities 

Commercial  Host 

Protect  user  data  on 

3rd  party  server 

Very  low 

Experience  Version 
management  at  server 

All,  Fast  upgrades 

Internal  Host 

EA  SW  certification 

Medium 

Complete  support  of  EA 

SW  and  servers 

All,  potential  delays  for 
certification 

Hot  Spot 

Allow  internet  on 
personal  networks 

Low 

Instructor  supplies 
personal  network 

Potential  latency  for 
network 

Instructor  Host 

Allow  personal 
networks 

Low 

Instructor  supplies 
network  and  application 

Potential  computer 
performance  issues 

Student  Host 

Allow  computers  to 
run  on  CD/DVD 

Low 

Need  to  burn  DVDs  with 
appropriate  client/server 
configuration  and  content 

No  multi-player  support, 
instructor  loses  access 

to  student  data 
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Migration  of  Development  to  3rd  Party  Support 

The  following  is  a  description  of  the  resources  and  efforts  for  migrating  the  EA  to  3rd  party 
support.  This  includes  identifying  support  staff  and  cost,  and  determining  process  and 
procedures  for  ticket  creation  and  processing. 


Support  Staff  and  Cost 

It  is  difficult  to  estimate  the  support  staff  requirements  and  cost  necessary  to  sustain  the 
Experience  Accelerator  as  it  is  still  under  active  development  and  has  not  been  productized. 
Currently,  it  is  difficult  to  discriminate  between  development  and  support  issues  as  new  features 
are  actively  being  developed  and  debugged,  and  the  code  is  being  refactored  in  the  process  to 
support  these  efforts.  Currently,  there  are  approximately  two  full-time  graduate  student 
equivalents  who  are  supported  at  approximately  20  hours  per  week  over  the  year  to  do  research 
and  to  update  and  support  the  EA  codebase,  update  the  tools  and  develop  new  experiences.  If 
20%  of  their  time  were  dedicated  to  support,  this  would  be  about  2.4  staff-months  of  effort 
annually. 

The  latest  major  development  task  that  is  related  to  support  is  the  migration  of  the  EA  from  Flash 
to  HTML5  which  provides  a  number  of  benefits  including  Section  508  compliance,  removal  of 
significant  firewall  issues,  portability  to  non-Flash  clients  (e.g.,  iPads,  iOS  devices)  and  significant 
improvements  in  the  capabilities  and  appearance  of  the  client  learner's  interface.  This 
development  effort  has  been  completed  which  will  reduce  future  support  effort  issues. 

The  intention  of  the  Experience  Accelerator  program  is  to  create  an  open  source  organization 
that  will  support  the  EA  over  time,  with  support  coming  from  volunteers  and  a  small  staff 
(probably  a  part-time  single  developer)  that  is  supported  by  user  donations.  There  has  been 
significant  interest  from  a  number  of  industrial  organizations  that  could  provide  the  funding  for 
continued  EA  infrastructure  and  experience  support. 


Processes  and  Procedures 

An  agile  development  process  is  being  used  by  the  EA  team  in  which  as  series  of  new  features 
and  capabilities  are  prioritized,  developed,  and  then  put  back  into  a  series  of  incremental 
releases.  Weekly  meetings  are  used  to  synchronize  the  team  on  the  state  of  development.  The 
integration  and  release  process  takes  place  at  Stevens  with  the  application  being  released  and 
accessible  on  www.experience-accelerator.org. 

During  the  past  Experience  Accelerator  Increments,  we  utilized  Trac  for  project  development, 
management  and  tracking.  And  Subversion  were  used  for  source  control. 

Trac  (http://trac.edgewall.org/)  is  an  open  source  enhanced  wiki  and  issue  tracking  system  for 
software  development  projects.  Trac  uses  a  minimalistic  approach  to  web-based  software  project 
management  and  allows  developers  freedom  in  determining  their  development  processes  and 
policies.  Trac  provides  an  interface  to  Subversion  and  Git  (or  other  version  control  systems),  and 
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integrated  Wiki  and  convenient  reporting  facilities.  Trac  allows  wiki  markup  in  issue  descriptions 
and  commit  messages,  creating  links  and  seamless  references  between  bugs,  tasks,  changesets, 
files  and  wiki  pages.  A  timeline  shows  all  current  and  past  project  events  in  order,  making  the 
acquisition  of  an  overview  of  the  project  and  tracking  progress  very  easy.  The  roadmap  shows 
the  road  ahead,  listing  the  upcoming  milestones. 

Subversion  is  an  open  source  version  control  system.  Founded  in  2000  by  CollabNet,  Inc.,  the 
Subversion  project  and  software  has  had  great  success  over  the  past  decade  and  has  enjoyed  and 
continues  to  enjoy  widespread  adoption  in  both  the  open  source  arena  and  the  corporate  world. 

Subversion  is  developed  as  a  project  of  the  Apache  Software  Foundation,  and  as  such  is  part  of  a 
rich  community  of  developers  and  users.  Subversion  "exists  to  be  universally  recognized  and 
adopted  as  an  open-source,  centralized  version  control  system  characterized  by  its  reliability  as 
a  safe  haven  for  valuable  data;  the  simplicity  of  its  model  and  usage;  and  its  ability  to  support  the 
needs  of  a  wide  variety  of  users  and  projects,  from  individuals  to  large-scale  enterprise 
operations."  (https://subversion.apache.org/) 

After  the  migration  to  FITML5  technology.  Team  Foundation  Service  was  used  for  FITML5  client 
source  control.  Server  source  control  was  moved  to  Bitbucket  and  SourceTree.  In  the  Experience 
Accelerator  Increment  4,  we  are  moving  toward  the  use  of  a  more  robust  configuration 
management  and  defect  ticketing  system,  most  of  which  will  be  used  in  an  open  source 
environment.  The  following  are  the  major  elements  of  this  system  in  the  future: 

•  Software: 

o  Project  Development,  Management  and  Tracking:  GitHub 
o  Source  control:  GitHub 
o  Flosting:  DigitalOcean 

•  Content: 

o  Dialog  files  -  Chat  Mapper 

o  Integration:  Artifact  Integrator  Tool,  upload  to  GitHub 
o  Software  upgrades:  versioned  release  trains  with  major  and  minor  releases 

•  Documentation: 
o  GitHub 

GitHub  (https://github.com)  is  a  web  based  version  control  repository  service.  GitHub  allows 
code  review,  project  management,  integrations,  community  management,  documentation  and 
code  hosting.  Projects  on  GitHub  can  be  accessed  and  manipulated  using  the  standard  Git 
command-line  interface  and  all  of  the  standard  Git  commands  work  with  it.  GitHub  also  allows 
registered  and  non-registered  users  to  browse  public  repositories  on  the  site.  Multiple  desktop 
clients  and  Git  plugins  have  also  been  created  by  GitHub  and  other  third  parties  that  integrate 
with  the  platform.  The  site  provides  social  networking-like  functions  such  as  feeds,  followers, 
wikis  (using  wiki  software  called  Gollum)  and  a  social  network  graph  to  display  how  developers 
work  on  their  versions  ("forks")  of  a  repository  and  what  fork  (and  branch  within  that  fork)  is 
newest. 
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We  are  currently  hosting  the  EA  on  Ubuntu  16.04.2  using  512MB  Ram  and  20GB  SSD  Disk  by 
DigitalOcean.com.  There  are  many  options  for  this  support  depending  on  the  specific  needs  with 
respect  to  data  storage,  memory,  processing,  bandwidth,  availability,  security  and  support. 


Update  to  EA  Documentation 

The  following  is  a  list  of  the  EA  documentation: 

•  Concept  of  Operations:  EA-CONOPS  vl.7,  August  2017 

•  Systems  and  Support  Specifications:  vl.O,  August  2017 

•  Architecture  and  Design:  EA-System-Architecture  vl.3,  August  2017 

•  Experience  Design:  EA_Design_Document_vl.8,  August  2017 

•  Tools:  EA-Tools  v3,  August  2017 

•  Instructor  Guide:  EA_lnstructor_Guide  v2.5,  Jun  2017 

•  Experience  Editor  User  Guide:  Experience_Editor_User_Guide_v0.9,  Apr  2017 


Software  Configuration  Items 

A  detailed  description  of  the  Experience  Accelerator  software  configuration  items,  and  runtime 
and  development  environment  Setup  is  shown  in  Appendix  A.  This  includes  URLs  for  all  open 
source  software  and  the  access  rights  for  each  software  application. 

Future  Work 


A  number  of  future  work  avenues  have  resulted  from  this  project.  One  aspect  of  the  project 
addressed  new  capabilities  and  improvements  to  the  UAV  experience.  These  no  doubt  will  be 
useful,  and  there  are  potential  other  new  capabilities  and  improvements  that  will  be  needed  over 
time. 

However,  we  have  received  feedback  from  DAU  instructors  that  they  may  be  more  interested  in 
more  tightly  focused  experiences  than  a  multi-phased  experience  that  tracks  an  entire 
development  program.  Thus,  it  would  be  useful  to  review  the  current  multi-phase  UAV 
experience  to  see  how  it  can  be  divided  into  smaller  experiences.  This  would  involve  learning 
objective  development  for  each  experience  module,  specification  of  initial  conditions  for  each 
module,  and  tuning  of  the  module  components  to  meet  the  learning  objectives  and  desired 
difficulty  level. 

With  the  completion  of  the  Section  508  compliance  work,  we  have  met  the  accessibility 
requirement.  Since  this  requirement  may  change  in  the  future,  there  may  be  additional  work 
needed  to  keep  the  SEEA  compliant  in  the  future.  More  interesting,  though,  is  research  aimed 
at  assessing  the  effectiveness  of  this  compliant  version  of  the  SEEA  with  respect  to  learners  who 
have  disabilities.  It  is  one  thing  to  meet  a  requirement.  It  is  another  to  meet  a  need. 


Report  No.  SERC-2017-TR-167 


50 


August  8,  2017 


While  we  have  made  progress  on  learning  evaluation  in  the  systems  engineering  domain,  much 
work  remains.  There  was  not  a  pilot  usage  of  the  SEEA  with  students  at  Defense  Acquisition 
University  during  the  project  as  originally  planned.  Rather,  there  was  a  demonstration.  We 
addressed  this  situation  by  conducting  learning  analysis  at  universities.  While  this  is  useful,  there 
are  differences  between  the  university  student  populations  and  the  student  population  at  DAU, 
which  is  more  focused  on  professional  development.  Clearly,  future  work  must  address  learning 
evaluation  in  the  context  of  courses  at  DAU.  With  the  increasing  data  from  EA  pilots  comes  the 
opportunity  to  develop  more  sophisticated  approaches  to  analyze  patterns  in  learning  and  have 
insight  into  the  capabilities  and  growth  of  the  learners.  This  is  a  very  fertile  area  for  future 
research. 

In  addition,  more  focus  is  needed  on  system  conception  and  design,  as  well  as  technical  trade¬ 
offs.  The  UAV  experience  starts  shortly  after  preliminary  design  review.  Thus,  it  does  not  capture 
the  initial  work  of  a  system  concept  and  analysis  of  alternatives.  The  technical  trade  study 
introduced  by  this  project  is  certainly  of  value,  but  it  has  not  yet  been  tested  in  a  course 
environment.  Thus,  we  need  to  evaluate  its  effectiveness  in  teaching  concepts  associated  with 
trade  studies. 

The  existing  experience  can  also  be  used  to  introduce  other  technical  trades.  For  instance,  the 
UAV  has  range  specified  as  a  key  performance  parameter.  The  underlying  technical  performance 
measures  related  to  range  include  weight  (sub-divided  by  sub-system),  drag  coefficient,  and 
propulsion  efficiency.  (Note  that  lift  coefficient  is  also  a  technical  performance  measure,  but  it  is 
assumed  constant  in  the  current  experience.)  There  are  two  additional  technical  performance 
measures  that  can  be  traded  against  range  -  loiter  and  payload  weight.  Loiter  is  the  time  allowed 
for  the  UAV  to  do  its  reconnaissance  and  other  work  over  a  target,  and  payload  weight  obviously 
is  the  allowed  weight  of  any  weapons  or  sensor  package  add-on.  These  are  modeled  in  the 
experience,  but  they  are  held  constant  and  the  learner  is  not  allowed  to  manipulate  them.  The 
experience  could  be  enriched  by  allowing  the  learner  to  trade  these  against  range  to  facilitate 
trade  study  learning. 

Furthermore,  other  experiences  can  be  developed  to  test  the  learning  effectiveness  of  the  overall 
SEEA  approach.  The  safety  engineering  experience  developed  in  conjunction  with  the  UK's 
Ministry  of  Defence  is  a  good  example  of  this  (Turner  et  al.,  2017).  We  have  also  discussed 
developing  an  experience  based  on  the  Wright  Brothers  flyer  that  was  developed  for  the  U.S. 
Army.  This  is  currently  a  paper  exercise  used  at  DAU  in  the  systems  engineering  curriculum. 
There  is  also  doctoral  work  in  the  area  of  the  assessment  of  systems  thinking  capabilities  using 
simulation.  A  number  of  Master's  projects  have  completed  the  design  work  for  the  development 
of  simulations  for  early  concept  phase  systems  engineering  skills  and  the  learning  of  engineering 
economics  through  an  entrepreneurial  experience. 

Future  work  also  involves  the  deployment  of  the  SEEA  at  DAU  and  other  institutions.  We  have 
laid  the  groundwork  for  this  with  tests  of  the  technology  platform  with  DAU  computers,  third- 
party  hosting,  and  deployment  guides.  However,  these  need  to  be  tested  in  an  actual 
deployment  environment. 
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Finally,  there  is  work  to  be  done  in  releasing  the  EA  and  its  tools  in  open  source  forms  and  the 
development  of  the  open  source  community  and  ecosystem.  There  is  significant  interest  in 
government,  industry  and  academia  to  support  these  efforts. 

Conclusion 


This  report  has  described  research  done  to  improve  and  validate  the  Systems  Engineering 
Experience  Accelerator  and  the  unmanned  aerial  vehicle  program  learning  experience  that  is 
deployed  using  the  SEEA  technology  platform. 

New  capabilities  were  added  in  terms  of  support  for  the  concepts  that  leaners  should  know  in  a 
major  systems  engineering  effort.  These  include  technical  trade  studies,  system  reliability,  and 
technical  debt.  In  addition,  a  number  of  improvements  were  made.  The  experience 
implementation  was  extended  from  critical  design  review  through  low-rate  initial  production  as 
originally  conceived.  The  simulation  was  tuned  to  help  ensure  realism.  The  non-player  character 
role  of  the  mentor  was  expanded  with  additional  resources  to  support  the  learner.  Finally, 
student  and  instructor  materials  were  updated  and  expanded. 

Since  the  SEEA  is  intended  for  use  at  Defense  Acquisition  University,  there  is  a  requirement  for 
compliance  with  federal  law  on  accessibility.  This  means  that  the  SEEA  must  comply  with  Section 
508  Amendment  to  the  Rehabilitation  Act  of  1973,  which  governs  computer  program  usage  as  it 
affects  disabled  employees  and  others.  The  SEEA  was  upgraded  from  an  Adobe  Flash  technology 
platform  to  FITML5  in  a  separate  project.  This  enabled  much  better  ways  to  comply  with  Section 
508  through  existing  compliance  checking  programs. 

One  of  the  most  important  aspects  of  the  project  was  validation  of  the  hypothesis  that  the  SEEA 
facilitates  systems  engineering  learning.  During  the  pilot  applications,  SEEA  was  unanimously 
praised  by  the  students  in  that  it  provided  an  opportunity  to  practice  the  skills  that  were 
illustrated  in  the  classroom.  Through  the  analyses  of  the  gathered  performance  and  behavioral 
data  during  multiple  pilot  uses,  it  is  safe  to  assume  that  the  EA  provides  the  capability  to  benefit 
the  learner's  systems  engineering  learning  process.  Learners'  performance  generally  improved 
between  multiple  run  of  the  experience.  Furthermore,  the  data  gathered  by  EA  also  provides 
insights  into  learner's  decision  making  and  information  analysis  process  making  future  training 
more  productive. 

Based  on  the  review  of  the  EA  with  the  DAU  faculty  in  June  2017,  the  use  of  the  EA  as  a  Workflow 
learning  tool  is  tentatively  planned  for  future  work  as  the  EA  is  not  planned  to  be  used  during  the 
Engineering  301  course.  There  are  numerous  possibilities  for  this  effort  which  leverages  the 
development  work  that  has  already  been  completed. 

Finally,  work  was  conducted  to  support  the  deployment  of  the  SEEA  at  Defense  Acquisition 
University.  Primarily,  this  consisted  of  developing  a  plan  for  third-party  support,  testing  the  SEEA 
application  on  DAU  computers  and  firewalls,  and  updating  documentation  for  SEEA  deployment 
and  usage. 
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At  project  initiation,  we  identified  the  risks  to  successful  conclusion  of  the  project.  These  risks 
were  managed  and  updated  during  project  execution,  and  the  risks  mitigation  plans  were 
successful  in  large  measure  as  described  below. 

Risk:  There  is  a  substantial  amount  of  work  that  needs  to  be  completed  in  the  remaining  two 
months  of  the  project,  in  particular  in  the  areas  of:  1)  completing  the  experience  throughout  the 
lifecycle,  2)  completing  the  trade-study,  3)  collecting  and  evaluating  learning,  4)  providing  the 
transition  path,  and  5)  ensuring  508  compliance  ability  to  validate  the  EA  at  the  DAU. 

•  Mitigation  strategy:  Focus  efforts  of  research  team  on  these  tasks  in  the  summer  months. 
Keep  close  track  on  progress  and  report  to  the  sponsor  and  make  trade-offs  on  any  issues 
that  might  arise.  Bring  in  additional  resources  as  necessary. 

•  Outcome:  The  mitigation  strategy  was  used  to  resolve  this  risk  successfully. 

Risk:  It  is  extremely  difficult  to  find  appropriate  measures  for  learning  in  the  selected  areas. 

•  Mitigation  strategy:  Review  and  model  research  on  measuring  learning  outcomes  in 
constructivist-based  learning  environments,  such  as  those  developed  with  case-based 
learning,  problem-based  learning,  project-based  learning,  and  discovery  learning 
methodologies,  in  order  to  determine  how  best  to  measure  critical  thinking,  problem¬ 
solving,  and  professional  skills  and  competencies,  measure  learner  perceptions  of 
learning  in  these  areas,  and  capture  the  EA  experience  of  some  learners  in  order  to 
capture  qualitative  evidence  of  learning,  as  exhibited  through  user  actions  and  strategies 
within  the  simulation. 

•  Outcome:  This  project  has  made  significant  progress  in  systems  engineering  learning 
evaluation.  Due  to  the  scale  of  the  research  area,  additional  research  is  needed  to  mature 
the  field. 

Risk:  There  are  difficulties  in  achieving  Section  508  Compliance. 

•  Mitigation  strategy:  There  is  substantial  existing  support  infrastructure  for  Section  508 
Compliance  of  HTML5  based  websites.  The  EA  will  leverage  these  by  transitioning  from 
Flash  to  FITML5  which  is  the  top  priority  research  item  in  the  EA  Tools  Part  3  project.  The 
effort  to  complete  this  conversion  has  been  estimated  and  is  not  a  significant  risk.  If  the 
project  falls  behind,  additional  resources  will  be  added  from  other  sources. 

•  Outcome:  The  conversion  to  FITML5  facilitated  the  Section  508  compliance  work. 
Updating  infrastructure  to  FITML5  touched  more  things  than  originally  planned,  though. 
This  introduced  schedule  pressure.  Flowever,  the  Section  508  work  was  completed. 

Risk:  The  ability  to  validate  the  EA  at  the  DAU. 

•  Mitigation  strategy:  One  option  is  to  substitute  this  as  a  capstone  for  a  couple  sections 
of  301  (approximately  60  students).  Another  option  is  to  make  this  available  on  the  DAU 
website  for  open  student  usage.  Additional  validation  provided  through  UAFI  classroom 
experiences. 
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•  Outcome:  There  was  a  demonstration  of  the  SEEA  to  instructors  at  DAU  on  June  1.  This 
replaced  the  previously  scheduled  pilot  class  usage.  Since  there  was  no  class  usage,  we 
deployed  the  SEEA  at  University  of  Alabama,  Huntsville,  and  at  AFIT  for  validation.  It  will 
potentially  be  used  at  Stevens  Institute  on  the  future. 

Risk:  The  likelihood  of  feature  creep  that  prevents  the  completion  of  an  adequate  set  of  tools  for 
to  provide  a  complete  environment  for  Experience  creation. 

•  Mitigation  strategy:  Utilize  agile  develop  practices  to  ensure  that  the  highest  value 
features  are  being  developed  at  all  times. 

•  Outcome:  This  risk  was  resolved  as  there  was  minimal  feature  creep  during  the  project. 

Risk:  Ensuring  that  there  are  graduate  student  staff  with  the  knowledge  and  capabilities  required 
for  effective,  efficient  tool  development  despite  gaps  in  funding. 

•  Mitigation  strategy:  Keep  current  graduate  students  engaged  as  much  as  possible  with 
stop  gap  funding  from  other  sources.  Determine  other  sources  of  students  or  software 
research  developers. 

•  Outcome:  This  risk  was  resolved  as  funding  for  the  EA  Tools  project  provided  interim 
funding  for  graduate  students. 

At  this  point,  the  SEEA  and  accompanying  UAV  experience  are  ready  to  be  used  in  a  number  of 
university  and  corporate  applications.  Clearly,  the  experience  will  need  to  be  tuned  to  meet  the 
needs  of  the  particular  student  population  in  terms  of  learning  objectives  and  difficulty  levels.  In 
addition,  other  experiences  have  been  developed  and  are  in  development  to  support  different 
learning  objectives  and  different  contexts.  We  are  pleased  to  report  that  the  SEEA  has  produced 
significant  interest  in  the  systems  engineering  education  and  training  community.  One  example 
of  this  was  the  reception  of  the  Best  Paper  Award  for  Systems  Engineering  Training  at  the  INCOSE 
(International  Council  on  Systems  Engineering)  2017  International  Symposium. 
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Appendix  C:  Accessibility  Requirements/Guidelines 


Section  508  (§  1194.22  Web-based  intranet  and  internet  information  and  applications.) 

(a)  A  text  equivalent  for  every  nontext  element  shall  be  provided  (e.g.,  via  "alt”,  "longdesc”, 
or  in  element  content). 

(b)  Equivalent  alternatives  for  any  multimedia  presentation  shall  be  synchronized  with  the 
presentation. 

(c)  Web  pages  shall  be  designed  so  that  all  information  conveyed  with  color  is  also  available 
without  color,  for  example  from  context  or  markup. 

(d)  Documents  shall  be  organized  so  they  are  readable  without  requiring  an  associated  style 
sheet. 

(e)  Redundant  text  links  shall  be  provided  for  each  active  region  of  a  server-side  image  map. 

(f)  Client-side  image  maps  shall  be  provided  instead  of  server-side  image  maps  except  where 
the  regions  cannot  be  defined  with  an  available  geometric  shape. 

(g)  Row  and  column  headers  shall  be  identified  for  data  tables. 

(h)  Markup  shall  be  used  to  associate  data  cells  and  header  cells  for  data  tables  that  have  two 
or  more  logical  levels  of  row  or  column  headers. 

(i)  Frames  shall  be  titled  with  text  that  facilitates  frame  identification  and  navigation. 

(j)  Pages  shall  be  designed  to  avoid  causing  the  screen  to  flicker  with  a  frequency  greater  than 
2  Hz  and  lower  than  55  Hz. 

(k)  A  text-only  page,  with  equivalent  information  or  functionality,  shall  be  provided  to  make  a 
web  site  comply  with  the  provisions  of  this  part,  when  compliance  cannot  be  accomplished  in 
any  other  way.  The  content  of  the  text-only  page  shall  be  updated  whenever  the  primary  page 
changes. 

(l)  When  pages  utilize  scripting  languages  to  display  content,  or  to  create  interface  elements, 
the  information  provided  by  the  script  shall  be  identified  with  functional  text  that  can  be  read 
by  assistive  technology. 

(m)  When  a  web  page  requires  that  an  applet,  plug-in  or  other  application  be  present  on  the 
client  system  to  interpret  page  content,  the  page  must  provide  a  link  to  a  plug-in  or  applet 
that  complies  with  §  1194.21(a)  through  (I). 

(n)  When  electronic  forms  are  designed  to  be  completed  on-line,  the  form  shall  allow  people 
using  assistive  technology  to  access  the  information,  field  elements,  and  functionality  required 
for  completion  and  submission  of  the  form,  including  all  directions  and  cues. 
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(o)  A  method  shall  be  provided  that  permits  users  to  skip  repetitive  navigation  links. 


(p)  When  a  timed  response  is  required,  the  user  shall  be  alerted  and  given  sufficient  time  to 
indicate  more  time  is  required. 

Notes  to  §  1194.22: 

1.  The  Board  interprets  paragraphs  (a)  through  (k)  of  this  section  as  consistent  with  the 
following  priority  1  Checkpoints  of  the  Web  Content  Accessibility  Guidelines  1.0  (WCAG 
1.0)  (May  5,  1999)  published  by  the  Web  Accessibility  Initiative  of  the  World  Wide  Web 
Consortium: 


Section  1194.22 

Paragraph 

WCAG  1.0 

checkpoint 

(a) 

1.1 

(b) 

1.4 

(c) 

2.1 

(d) 

6.1 

(e) 

1.2 

(f) 

9.1 

(g) 

5.1 

(h) 

5.2 

(i) 

12.1 

(j) 

7.1 

(k) 

11.4 

2.  Paragraphs  (I),  (m),  (n),  (o),  and  (p)  of  this  section  are  different  from  WCAG  1.0.  Web  pages 
that  conform  to  WCAG  1.0,  level  A  (i.e.,  all  priority  1  checkpoints)  must  also  meet 
paragraphs  (I),  (m),  (n),  (o),  and  (p)  of  this  section  to  comply  with  this  section.  WCAG  1.0 
is  available  at  http://www.w3.org/TR/1999/WAI-WEBCONTENT-19990505. 

Web  Content  Accessibility  Guidelines  (WCAG) 

The  Web  Content  Accessibility  Guidelines  (WCAG)  are  part  of  a  series  of  web  accessibility 
guidelines  published  by  the  Web  Accessibility  Initiative  (WAI)  of  the  World  Wide  Web  Consortium 
(W3C),  the  main  international  standards  organization  for  the  internet.  They  consist  of  a  set  of 
guidelines  for  making  content  accessible,  primarily  for  people  with  disabilities,  but  also  for  all 
user  agents,  including  highly  limited  devices,  such  as  mobile  phones.  The  current  version,  WCAG 
2.0,  was  published  in  December  2008  and  became  an  ISO  standard,  ISO/IEC  40500:2012  in 
October  2012.  Numerous  nations  are  using  the  WCAG  standards  as  the  basis  for  their  accessible 
regulations. 

WCAG  is  based  on  the  following  accessibility  principles: 

Perceivable 

Information  and  user  interface  components  must  be  presentable  to  users  in  ways  they  can 
perceive. 
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•  Guideline  1.1:  Provide  text  alternatives  for  any  non-text  content  so  that  it  can  be  changed 
into  other  forms  people  need,  such  as  large  print,  braille,  speech,  symbols  or  simpler 
language. 

•  Guideline  1.2:  Time-based  media:  Provide  alternatives  for  time-based  media. 

•  Guideline  1.3:  Create  content  that  can  be  presented  in  different  ways  (for  example 
simpler  layout)  without  losing  information  or  structure. 

•  Guideline  1.4:  Make  it  easier  for  users  to  see  and  hear  content  including  separating 
foreground  from  background. 

Operable 

User  interface  components  and  navigation  must  be  operable. 

•  Guideline  2.1:  Make  all  functionality  available  from  a  keyboard. 

•  Guideline  2.2:  Provide  users  enough  time  to  read  and  use  content. 

•  Guideline  2.3:  Do  not  design  content  in  a  way  that  is  known  to  cause  seizures. 

•  Guideline  2.4:  Provide  ways  to  help  users  navigate,  find  content,  and  determine  where 
they  are. 

Understandable 

Information  and  the  operation  of  user  interface  must  be  understandable. 

•  Guideline  3.1:  Make  text  content  readable  and  understandable. 

•  Guideline  3.2:  Make  web  pages  appear  and  operate  in  predictable  ways. 

•  Guideline  3.3:  Help  users  avoid  and  correct  mistakes. 


Robust 

Content  must  be  robust  enough  that  it  can  be  interpreted  reliably  by  a  wide  variety  of  user 
agents,  including  assistive  technologies. 

•  Guideline  4.1.:  Maximize  compatibility  with  current  and  future  user  agents,  including 
assistive  technologies. 

Relationship  of  WCAG  2.0  and  Section  508 

The  following  is  the  general  relationship  between  WCAG  2.0  and  Section  508: 

•  WCAG  2.0  harmonizes  with  Section  508 

•  WCAG  2.0  Level  AAA  is  Section  508  compliant 

•  WCAG  2.0  Level  AA  is  a  reasonable  standard  to  strive  for 

•  WCAG  2.0  Levels  are  prioritized  based  on  time,  money,  audience,  importance 

•  Priority  1  =  Level  A  is  required,  you  must  do  it 

•  Priority  2  =  Level  AA  is  preferred,  it  would  be  great  if  you  did  it 

•  Priority  3  =  Level  AAA  is  optional,  it  would  be  nice  to  have  (but  required  for  government 
use) 

•  WCAG  2.0  Level  AAA  is  equivalent  to  Section  508  compliance,  it  is  the  government 
standard 
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Appendix  D:  EA  Runtime  and  Development  Environment  Setup 

The  following  is  a  detailed  description  of  the  Experience  Accelerator  software  configuration 
items,  access  rights,  and  runtime  and  development  environment  Setup.  This  includes  URLs  for 
all  open  source  software. 

Software  Applications 

The  following  is  a  list  of  the  software  used  in  the  development  and  deployment  of  the  Experience 
Accelerator  (EA).  The  software  is  categorized  as  being  used  in  the  Server,  Client  or  Development. 
For  deployment.  Server  and  Client,  the  only  necessary  proprietary  software  is  the  Adobe  Flash 
Player.  A  conversion  to  HTML5  would  remove  this  dependency  and  allow  the  EA  to  be  accessed 
on  Apple  mobile  devices.  For  development,  the  only  proprietary  tool  used  is  Chatmapper  for 
GUI-based  dialog  creation  and  management. 

Server: 

•  WampServer  (http://www.wampserver.com/en/):  Open  Source  and  Free  under  GNU 
General  Public  License.  -  http://www.gnu.org/licenses/gpl.html 

•  SystemDynamics  (http://sourceforge.net/proiects/svstem-dynamics):  Open  Source  and 
Free  under  GNU  General  Public  License  version  2.0  (GPLv2) 
http://www.gnu.Org/licenses/old-licenses/gpl-2.0.html 

•  JFreeChart  (http://www.jfree.org/jfreechart/):  Open  Source  and  Free  under  GNU  Lesser 
General  Public  License  -http://www.gnu.org/licenses/lgpl.html 

•  Experience  Accelerator:  Developed  under  SERC  DAU  contract,  no  parties  have  claimed 
patent  rights  and  desire  the  software  to  be  available  under  open  source  licensing. 

o  Simulation:  Georgia  Tech  development 
o  System:  Stevens  and  Purdue  development 


Client: 

•  Browser:  Proprietary  and  open  source  versions  exist  (Microsoft  Edge,  Google  Chrome, 
Mozilla  Firefox  and  Apple  Safari  supported.  Mobile  versions  of  these  browsers  are 
supported.) 

Development: 

•  Visual  Studio  2015:  Community  Edition  Free.  -  https://www.visualstudio.com/ 

•  Eclipse  SDK:  Open  Source  and  Free  under  Eclipse  Public  License. 
-  http://www.eclipse.org/org/documents/epl-vlO.php 

•  Chat  Mapper:  $495  for  commercial  license.  Works  developed  using  Chat  Mapper  are 
allowed  to  be  distributed  under  commercial  license  -http://www.chatmapper.com/eula/. 
Source  Code  License  available. 

•  EA  Tools:  Developed  under  SERC  core  contract,  no  parties  have  claimed  patent  rights  and 
desire  the  software  to  be  available  under  open  source  licensing. 


Environment  Setup 
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Steps  for  building  and  running  the  experience  accelerator  on  Windows  from  source: 

1.  Eclipse  Classic 

a.  Download  -  http://www.eclipse.org/downloads/ 

b.  Extract  somewhere 

c.  eclipse.exe  is  used  to  run  (create  shortcut  for  desktop  if  wanted) 

2.  Download  and  install  Java:  http://www.java.com/en/download/index.jsp 

3.  Download  and  install  FlashDevelop:  http://www.flashdevelop.org/ 

4.  Download  and  install  visual  C++ 

a.  32bit:  http://www.microsoft.com/en-us/download/details.aspx?id=8328 

b.  64bit:  http://www.microsoft.com/en-us/download/details.aspx?id=13523 

5.  Download  and  install  WampServer:  http://www.wampserver.com/en/ 

6.  Run  WampServer.  A  W  should  appear  in  the  icons  in  the  lower  right  of  the  taskbar.  The 
W  should  go  from  red  to  orange  to  green. 

a.  You  may  have  to  right  click  and  "run  as  administrator" 

b.  You  may  have  to  close  other  programs  that  use  port  80  such  as  skype  before 
running 

c.  If  a  W  still  does  not  appear  after  trying  the  above,  wait  a  few  minutes  and  then  try 
step  7. 

7.  After  the  W  is  green,  go  to  127.0.0.1/phpmyadmin/  in  a  browser 

8.  If  prompted  for  a  username  and  password,  enter  root  as  the  username  and  no  password 

9.  Once  in  phpmyadmin,  create  a  new  database. 

a.  Go  to  Databases ->  create  database 

b.  Name  it  expaccdev 

c.  Use  the  default  type  "Collation" 

10.  Go  to  the  newly  created  database  and  go  to  Privileges  ->  Add  user 

a.  username:  expaccdev 

b.  host:  any  host 

c.  password:  stevensl23 

d.  Database  for  user:  Grant  all  privileges  on  database  "expaccdev" 

e.  Leave  Global  Privileges  blank 

11.  Create  another  user  on  the  database  with: 

a.  user  name:  expaccdev 

b.  host:  localhost 

c.  password:  stevensl23 

d.  Database  for  user:  Grant  all  privileges  on  database  "expaccdev" 

e.  Leave  Global  Privileges  blank 

12.  Import  the  database.  Go  to  lmport->Choose  File->choose  [EA_source_root]/ 
server/db/expaccdev.sql.  Press  Go. 

13.  Setup  Eclipse  Project 

a.  Run  Eclipse 

b.  File->lmport->General->Existing  Projects  into  Workspace 

c.  Click  next 

d.  Click  Browse  for  Select  root  directory:  [EA_source_root]/server 

e.  Click  Finish 
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f.  Select  src.ea.server.TCP_Server.java 

g.  Run  (Green  Arrow) 

i.  Choose  Java  Application  if  asked 

ii.  Choose  TCP_Server  if  asked 
14.  Run  client  (FlashDevelop  project) 

a.  Double  click  [EA_source_root]/client/Client.as3proj 

b.  Open  src.ea.net. Connection. as 

i.  Set  USE_LOCAL_HOST  =  true 

ii.  Set  USE_PORT_80  =  false 

c.  Test  Project  (Blue  play  button) 

i.  May  have  to  first  do  Project->Clean  Project 

Server  Deployment 
Setup  LAMP  Server 

Using  DigitalOcean:  https://www.digitalocean.com/community/tutorials/how-to-install-linux- 
apache-mysql-php-lamp-stack-on-ubuntu 

Setup  Host  for  Client 

Bluehost,  GoDaddy,  etc 

For  File  Transferring 

FileZilla:  https://filezilla-project.org/download.php?type=client 

Build  for  Server  Deployment 

To  deploy  to  a  LAMP  (Linux  Apache  MySQL  Phpmyadmin)  server,  follow  the  steps  below: 

1.  Create  a  Server.jar  file 

a.  Open  server  project  in  Eclipse 

b.  Go  to  File->Export 

c.  Select  Java->Runnable  JAR  file 

d.  Click  Next 

e.  Launch  Configuration:  TCP_Server  -  server 

f.  Choose  an  export  destination.  Name  the  file  Server.jar 

g.  Library  handling:  Package  required  libraries  into  generated  JAR 

h.  Click  Finish 

2.  Copy  [EA_source_root]/server  to  a  directory  on  the  LAMP  server  [EA_server_root] 

3.  Copy  the  generated  Server.jar  file  to  [EA_server_root]/server 

4.  In  a  browser,  go  to  [server  ip  addressj/phpmyadmin/ 

5.  If  prompted  for  a  username  and  password,  enter  root  as  the  username  and  no  password 

6.  Once  in  phpmyadmin,  create  a  new  database. 

a.  Go  to  Databases ->  create  database 

b.  Name  it  expaccdev 

c.  Use  the  default  type  "Collation" 

7.  Go  to  the  newly  created  database  and  go  to  Privileges  ->  Add  user 

a.  username:  expaccdev 
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b.  host:  any  host 

c.  password:  stevensl23 

d.  Database  for  user:  Grant  all  privileges  on  database  "expaccdev" 

e.  Leave  Global  Privileges  blank 

8.  Create  another  user  on  the  database  with: 

a.  user  name:  expaccdev 

b.  host:  localhost 

c.  password:  stevensl23 

d.  Database  for  user:  Grant  all  privileges  on  database  "expaccdev" 

e.  Leave  Global  Privileges  blank 

9.  Import  the  database.  Go  to  lmport->Choose  File->choose  [EA_source_root]/ 
server/db/expaccdev.sql.  Press  Go. 

10.  On  the  LAMP  server,  navigate  to  [EA_server_root]/server 

11.  To  run  the  server,  use  the  command:  java  -jar  Server.jar 

12.  On  the  web  server  that  will  host  the  client,  create  a  directory  [EA_hosted_client] 

13.  Create  a  Client. swf  file 

a.  Double  click  [EA_source_root]/client/Client.as3proj 

b.  Open  src.ea.net. Connection. as 

i.  Set  USE_LOCAL_HOST  =  false 

ii.  Set  USE_PORT_80  =  false 

iii.  Scroll  down  to  the  Connect  function 

iv.  Set  the  server  ip  address  in:  var  urLString  =  "ip  address" 

c.  Go  to  Project->Build  Project 

i.  May  have  to  first  do  Project->Clean  Project 

14.  Copy  [EA_source_root]/client/bin/Client.swf  to  [EA_hosted_client] 

15.  Copy  [EA_source_root]/currentbuild/index.html  to  [EA_hosted_client] 

16.  To  run  the  client  in  a  browser,  go  to  [client  web  server  ip  address  or 
url]/[EA_hosted_client] 

Update  Server 

1.  Create  an  updated  Server.jar  file 

a.  Open  server  project  in  Eclipse 

b.  Open  src.ea.server.TCP_Server.java 

c.  Increment  CURRENT_CLIENT_VERSION  (should  match  client  version) 

d.  Go  to  File->Export 

e.  Select  Java->Runnable  JAR  file 

f.  Click  Next 

g.  Launch  Configuration:  TCP_Server  -  server 

h.  Choose  an  export  destination.  Name  the  file  Server.jar 

i.  Library  handling:  Package  required  libraries  into  generated  JAR 

j.  Click  Finish 

2.  On  LAMP  server  (e.g.,  DigitalOcean  server),  stop  the  experience  accelerator  if  running 
(Ctrl+C) 

3.  Replace  any  updated  files  on  the  server  (e.g.,  Server.jar)  using  FileZilla 
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4.  Make  any  needed  changes  to  database  through  phpmyadmin 

5.  Restart  experience  accelerator  on  LAMP  server  using:  java  -jar  Server.jar 


Update  Client 

1.  Replace  html/css/javascript  files  on  host  (Bluehost,  GoDaddy,  etc)  with  newly  developed 
files 

Pre-built  Local  Version 

A  local  running  version  of  the  experience  accelerator  exists  for  easier  running  in  demos 
where  internet  access  is  not  available. 

Setup  Database  for  Local  Version 

1.  Download  and  install  visual  c++ 

a.  32bit:  http://www. microsoft.com/en-us/down load/details. aspx?id=8328 

b.  64bit:  http://www. microsoft.com/en-us/down load/details. aspx?id=13523 

2.  Download  and  install  WampServer:  http://www.wampserver.com/en/ 

3.  Run  WampServer.  A  W  should  appear  in  the  icons  in  the  lower  right  of  the  taskbar.  The 
W  should  go  from  red  to  orange  to  green. 

a.  You  may  have  to  right  click  and  "run  as  administrator" 

b.  You  may  have  to  close  other  programs  that  use  port  80  such  as  skype  before 
running 

c.  If  a  W  still  does  not  appear  after  trying  the  above,  wait  a  few  minutes  and  then  try 
step  7. 

4.  After  the  W  is  green,  go  to  127.0.0.1/phpmyadmin/  in  a  browser 

5.  If  prompted  for  a  username  and  password,  enter  root  as  the  username  and  no  password 

6.  Once  in  phpmyadmin,  create  a  new  database. 

a.  Go  to  Databases ->  create  database 

b.  Name  it  expaccdev 

c.  Use  the  default  type  "Collation" 

7.  Go  to  the  newly  created  database  and  go  to  Privileges  ->  Add  user 

a.  username:  expaccdev 

b.  host:  any  host 

c.  password:  stevensl23 

d.  Database  for  user:  Grant  all  privileges  on  database  "expaccdev" 

e.  Leave  Global  Privileges  blank 

8.  Create  another  user  on  the  database  with: 

a.  user  name:  expaccdev 

b.  host:  localhost 

c.  password:  stevensl23 

d.  Database  for  user:  Grant  all  privileges  on  database  "expaccdev" 

e.  Leave  Global  Privileges  blank 

9.  Import  the  database.  Go  to  lmport->Choose  File->choose  [EA_source_root]/ 
server/db/expaccdev.sql.  Press  Go. 
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Update  Local  Version 

1.  Create  an  updated  Server.jar  file 

a.  Open  server  project  in  Eclipse 

b.  Go  to  File->Export 

c.  Select  Java->Runnable  JAR  file 

d.  Click  Next 

e.  Launch  Configuration:  TCP_Server  -  server 

f.  Choose  an  export  destination.  Name  the  file  Server.jar 

g.  Library  handling:  Package  required  libraries  into  generated  JAR 

h.  Click  Finish 

2.  Copy  the  updated  contents  of  [EA_source_root]/server  to  [EAJocalj/server 

3.  Copy  the  generated  Server.jar  file  to  [EAJocalj/server 

4.  Create  an  updated  the  client  files 

5.  Copy  the  new  client  files  to  [EA_local]/EA  to  overwrite  the  old  ones 

Run  Local  Version 

1.  Run  WampServer 

2.  Host  the  client  folder  as  a  localhost  website 

3.  Go  to  [EAJocal]  and  double  click  runServer.bat  to  launch  the  java  server 

Add  a  User 

Instructions  for  adding  a  new  user  account  on  the  experience  accelerator. 

1.  Navigate  to  [EA_source_root]/server/FS_root/users 

2.  Create  a  new  directory  matching  the  new  username 

3.  In  a  browser,  go  to  127.0.0.1/phpmyadmin 

a.  Make  sure  to  run  WampServer  first 

4.  Go  to  Databases->expaccdev->users 

5.  Click  Insert 

6.  Fill  in  info  for  new  user 

a.  Username:  should  match  created  folder 

b.  Pwd:  password 

7.  Click  Go 

8.  Run  the  experience  accelerator  and  login  as  newly  created  user 

9.  Click  Abort  (needed  to  fill  user  directory  for  first  time) 

Add  pdf  Artifact 

Add  a  static  pdf  to  the  experience  accelerator  for  users  to  see  in  the  virtual  file  system. 

1.  Move  the  created  pdf  file  to  [EA_source_root]/  server/FS_root/common/stat-artifacts 

2.  Go  to  [EA_source_root]/server/FS_root/common/eaFileSystem/phases 

3.  Open  the  xml  file  for  the  phase  when  the  file  should  become  visible 

4.  Add  an  entry  to  the  xml:  <File  name="display  page  name"  path="virtual  path" 
pageName="page  name"> 

a.  Display  page  name  is  the  name  that  will  appear  in  the  experience  accelerator  file 
explorer 
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b.  Virtual  path  is  where  the  file  will  appear  when  using  the  file  explorer  in  the 
experience  accelerator  client. 

c.  Page  name  should  match  the  page  name  from  step  1. 

Update  Simulator 

To  update  the  simulator  used  by  the  experience  accelerator,  replace  any  updated  files  in 
[EA_source_root]/  server/FS_root/common/ - /local_sim 
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